Hi everyone, I am currently working on a benchmarking project that aggregates and compares regional electricity tariff structures against standardized public energy consumption models, similar to what we see in Western Australia’s open utility datasets. While mapping external municipal billing schemas into a unified JSON format, I ran into an issue handling 14-digit reference consumer lookups on this website, where automated retrieval frequently stalls due to unpredictable response latency and dynamic session header mismatches.
The secondary challenge is that duplicate consumer reference identifiers across different sub-divisions fail silently without throwing standard HTTP error codes, making automated data validation and ingestion pipelines fail downstream. Has anyone here implemented an elegant fallback strategy or queue architecture when normalizing inconsistent external public billing endpoints into an open data schema, especially when dealing with legacy query interfaces that lack structured REST endpoints and proper error schemas?
Any insights on schema normalization and retry patterns for public utility data would be greatly appreciated.