Excerpt from the actual review document (Health NZ Financial Management Review, [0]):
Notably, one major issue was through a significant reliance on the use of an Excel file to manage the consolidated financials of the organisation. This spreadsheet was the primary data file used by HNZ to manage its financial performance. It consolidated files from each district into a single spreadsheet, and key reports, such as the monthly finance report, were produced from it. The use of an Excel spreadsheet file to track and report financial performance for a $28bn expenditure organisation raises significant concerns, particularly when other more appropriate systems are present on the IT landscape.
This Excel file is flawed in that:
• Financial information was often 'hard-coded,' making it difficult to trace to
the source or have updated data flow through.
• Errors such as incorrectly releasing accruals or double-up releases were not
picked up until following periods.
• Changes to prior periods and FTE errors in district financial reporting Excel
submissions, would not flow through to consolidated file.
• The spreadsheet can be easy to manipulate information as there is limited
tracking to source information where information is not flowing directly from
accounting systems.
• It is highly prone to human error, such as accidental typing of a number or
omission of a zero.
So actually top level planning/reporting is done by everything flowing into one big spreadsheet - as crappy as that is, thats pretty common. Thats not the same as "managed finances with single Excel spreadsheet". Like, they do have other systems.
25 years ago a small Norwegian company in oil industri replaced a big Excel that it were using to manage its inventory with Access database and discovered that they have like extra 200K USD in equipment.
The reason for the saving was enforcement of constrains in various Access tables so a subtle entry error in Excel was discovered.
The transition to DB file was done by a single person in a month who had not worked previously with DB (however he was young and talented programmer I must admit).
And the Access DB in a single file with its UI was quite similar to Excel so guys maintaining it quickly learned it and really liked it.
So it is still puzzles me why such single file DB with nice UI are not more popular? Why it is always Excel?
Someone builds an access database and it works for a bit but then they leave and no-one else knows how it works so they start exporting the data to excel instead. Access is all about restrictions (the form has these fields; the data is structured like so; input will be validated by these rules) ... Restrictions deliver some value but are brittle - its about as hard to maintain as any other custom built app would be.
as an anecdote, my own mother, who studied french in university and works in education, has built access DBs for places she's worked before. It seems its a very approachable and beginner friendly UI for non-programmers, somehow.
> why such single file DB with nice UI are not more popular?
Because it needs skills that often aren't immediately available whereas a lot of people have enough Excel experience to lash up even a moderately complex spreadsheet.
At many places there's also some nasty politics around siloing where a database is an IT System so it has to be owned by the centralized IT organization, operated by a DBA team, have a budget, be audited by security, etc. and half of those groups look down on Access and want to architect for job security by picking some Oracle/SAP product where even if you have an unlimited budget you are looking at years of guaranteed delay before you can use it with a significant risk of failure.
In a fair number of places that I've worked the finance and IT departments have a fairly substantial level of enmity which also discourages use of anything which involves them. Meanwhile, everyone still needs to do their jobs and there's a business analyst with decent Excel chops who will have something you can use “temporarily”. A few years pass, then a few more, and now everything lives in Excel as the calcified scar tissue of broken official processes…
> At many places there's also some nasty politics around siloing where a database is an IT System so it has to be owned by the centralized IT organization, operated by a DBA team, have a budget, be audited by security, etc. and half of those groups look down on Access and want to architect for job security by picking some Oracle/SAP product where even if you have an unlimited budget you are looking at years of guaranteed delay before you can use it with a significant risk of failure.
A space seemingly ripe for Line Of Business Applications As A Service: LOBAAAS :-)
TBH, the whole SaaS took off because of the ability for individual departments to just get their resources using nothing but a company credit card.
Excel is really easy to use for smaller data collection and manipulation. There's no input validation, signing and other complicated design decisions needed to start an excel spreadsheet. There's alot of tools out there for easy integration. So its not something you need to switch away from once you have rather static requirements. Billions are just numbers that fit inside a cell just fine until someone complains.
FWIW, having a human regularly count the equipment would have also uncovered the error. As well as other interesting reasons that discrepancies arise, such as theft. Always important remember that these systems are tools, not the truth, and have to be reconciled to the truth on a regular basis.
>So actually top level planning/reporting is done by everything flowing into one big spreadsheet - as crappy as that is, thats pretty common.
Yeah, everywhere I've worked that's common, even when the underlying data is managed much more successfully, the top folks really want everything on a spreadsheet.
Replacing this excel sheet with a 'proper' system would cost between $5million and $20million probably, depending on what sort of consultancy delivered it
And would be inflexible, and maybe still wrong, and not necessarily more transparent... So it may mean several extra million dollars per year in ongoing maintenance if things turn out typically...
Excel has many issues, but the cost of replacing it is surprisingly high.
>And would be inflexible, and maybe still wrong, and not necessarily more transparent...
I'm going through this right now. Not anywhere in this scale, but just in terms of deciding whether to move an order-tracking/accounting system I've built in Excel to a "real" database.
The database I am considering (Panorama X for Mac) is quite inexpensive, I've heard very good things about it, and has a spreadsheet-like UI. However, I've used Excel enough to know that I haven't tapped more than a small fraction of its ability, especially things like Power Query. As much as I loathe VBA, what if the cost of moving to a "real" database isn't the up-front cost, but the longer-term inflexibility of Panorama (and, pretty much, anything else in my price range) compared to the beast that is Excel?
>[1] Except the one you want the answer to at a given moment, amirite fellas?
Everytime I relearn how to do a vlookup, none of the examples online do what I want. It's like they are trying to do something completely opposite of what I want. It's super weird, because what I want to do is always a pretty common thing to need done.
No. This is not a common practice. An Accounting / ERP system with full audit trails and data entry compliance is. Exporting data from such a system into Excel for analysis is. But Excel as the [re-reads] Primary data file is not a practice at all.
I've worked on dozens of Excel to ERP migrations. Excel is way more common as the primary data source including financial data. If you only look at financial data by volume (and not count) maybe ERP win, but I wouldn't be sure.
> and was used for “consolidation, journals, business-critical reporting, and analysis.”
Not sure what 'journals' is supposed to mean here but “consolidation, business-critical reporting, and analysis.” does not mean operations day to day use, it means high level management reporting
My position is that the article (or the consultants who are trying to sell the ERP) are exaggerating the role of the spreadsheet.
If the Excel file is just a snapshot of the financial situation and it doesn't feed back into the main system, I find it harder to see the issue. Essentially it sounds like a manager takes the Excel file, makes some small changes, writes some equations, produces tables/graphs and then writes a report.
Maybe there is an error in the reporting, but it can always be corrected. The idea isn't to explain where $16.129626356 bn dollars went, it's to explain where ~$16 bn dollars went.
I can think of many processes just off the top of my head where commercial software outputs something like a CSV file, I add my data to it, then generate a report. I don't think the process is insane at all. We catch errors by predicting the data to be entered and flagging if off by 10%, and then also looking at the distribution of the data entered (it should look like a skewed bell curve). We then have a stand-up meeting where we review the outputs and then have a process for later correction. Of course it's possible for errors to creep in, but the errors are very unlikely and have somewhat limited scope.
My point is that the situation in NZ seems entirely resolvable, and they shouldn't just throw it out if it was somewhat working. The whole "Excel" thing seems like a nothing-burger.
If you spend a billion and a couple years redoing the generally working system to (maybe!) increase the number of significant digits on a dashboard for no good reason, you may have screwed up even more.
I think the important part that's not clearly portrayed in the OP article is that the "one Excel file" is actually not a single file used across the whole organisation, but rather a sort of master overview file that "consolidated files from each district into a single spreadsheet".
I don't think that fact is made very clear in the OP article.
Some of those defects seem as if they could be used to improve spreadsheet software more than they mean that spreadsheet software is intrinsically inappropriate.
The amount of issues I have seen in the last ~20 years caused by "off by one" type of errors in Excel is insane.
Few examples:
- incorrect schedule for the electricity grid for an entire country
- incorrect assessment of an airport use for an airline (causing few millions USD loss in revenue)
- incorrect financial position assessment for a mine (resulting incorrect deciosion to optimize the wrong business process, not sure about how much they lost)
Making illegal states unrepresentable is a concept that benefits programming langues and business processes alike.
You’re only looking at one side of it. Consider how much value that excel adds, and how many use cases it enables. It’s incalculable. It’s always easy to poke at excel’s issues, but errors are almost always a skill issue.
From a financial perspective, there are many ways you could enforce checks to ensure the model is balanced - it just takes time. Data entry can be an issue, but you can automate that too. The deloitte report is saying the health department should benefit from adopting a gigantic erp system but you could get 90% of the benefit by employing a couple people that really know what they’re doing.
You could say that excel should allow those things to happen but the flexibility is precisely what makes it so valuable.
The people writing the reports are consultants. Consultants recommend things that benefit consultants. In their case, a multi year process and tens of millions of dollars trying to install ERP software is more a windfall for the vendors and not the companies.
Yep. I was going to say the same thing. Installing some $10M+ software costs the company more than just the install and ongoing licensing costs. You have to have an entire new IT team to manage it and staff to understand and so on. There are probably just as many chances of having a big issue.
That assumes that Better Excel would somehow not look like Excel and would lose its benefits of being easy. My assumption is that Better Excel will still look like Excel, in the same way that Better iOS will also look like iOS.
Sure but to compare it to tech, this is why we build tooling like linters and typed languages so that we don’t lose millions of dollars because we needed the average developer to implement a feature. I imagine in heath care it’s even more skewed because a SW eng can see the compiler output and make changes but the person entering the data isn’t (for example) the nurse that’s using the needles to be accounted for
There are indeed tools like this available for Excel. And there are also whole practices within the big accountancy firms, and independent firms which focus on providing "audit" of Excel files like this, which are used commonly where there are financial firms involved in a project with big money at stake (lenders, financial investors, etc). But in my experience these tools and services aren't often used for the spreadsheets which are used for day to day corporate-type management (I would count government as part of that).
Say nothing of many crisis were avoided because somebody was able to pull up a local excel file despite a central service being down or go through their historical files and sniff out a discrepancy. Workflows like that are often impossible or require coordinating with a vendor in another time zone or a bunch of slow cross team communication and ticket submissions with the sort of minimum effort "we'll run a DB in the cloud and slap a JS front end built with a bunch of questionable but flashy libraries and ignore all the edge cases because I'll be working somewhere else in 2yr" solutions that the median software developer hive mind will implement if left to its own devices.
Do you have evidence that _any_ other software would not be prone to this kind of errors, taking into account the fact that at the scale of dozens of billions of dollars, the numbers you mentioned seem like rounding errors (except for the first one which would deserve a reference)?
>The amount of issues I have seen in the last ~20 years caused by "off by one" type of errors in Excel is insane.
Generally those aren't a fault of excel, but a fault of someone doing something dumb in excel. People do dumb stuff in every problem and it's nearly impossible to prevent it.
Don't forget Reinhart-Rogoff, which ties into the other thread on academic fraud/nonreproduction, and arguably did huge amounts of economic damage: https://www.bbc.co.uk/news/magazine-22223190
Imagine how many errors occurred before we had digital spreadsheet technology.
That said, I’ve always thought there was a product that sits somewhere in between Excel and full blown custom software that provides some of the controls we need while still being and build able by someone with low/average technical skills
FileMaker seems to cover this (admittedly my interactions with it was in high school, which was last millennium), it's a database app but there's a lot of "make your own UI" parts in it, so you can create custom UIs (and wizards) for your use cases.
I suppose MS Access offers this too, although FM feels more user-friendly, MS Access felt like Internet Explorer 4, where an error dialog would pop-up for every little JavaScript error (disclaimer: this opinion is from 25 years ago).
No you're right on the money in my opinion. I have very fond memories of spinning up apps in MS Access that were quick and dirty but extremely powerful, and able to be maintained/modified by folks that were not professionally trained software engineers. I think there's a missing area in the market for something exactly like that again.
The hive mind really hated (hates?) MS Access and I don't know why.
I built a mini ERP for my father's company when I was 16. Started out as an excel spreadsheet, but then added inventory, accounting, and printing contracts and reports.
He kept using it until retirement and was very happy with it. I could learn & apply SQL. Win-win.
I've done that over the last several years, building an order-tracking/accounting system for myself in Excel. (Inventory and fulfillment is handled by commercial software.) I'm mulling over whether to move to a "real" database.
I'm on a Mac so can't move to Access. I've thought about FileMaker, but am considering Panorama X because unlike FileMaker it reportedly allows undoing almost anything, while FileMaker is like the typical database in a record commit not being undoable.
(Yes, I know that not allowing such is good practice, which is why I am not a database admin.)
I've heard good things about Panorama X, and it has a spreadsheet-like UI. However, I've used Excel enough to know that I haven't tapped more than a small fraction of its ability, especially things like Power Query. As much as I loathe VBA, what if the cost of moving to a "real" database isn't the up-front cost or conversion time, but the longer-term inflexibility of Panorama (and, pretty much, anything else in my price range) compared to the beast that is Excel?
I've not used it, but I thought Airtable occupied this spot. It seems to have a few open-source clones if you want to run it locally. I don't know if they are any good.
> The spreadsheet-using agency is Health New Zealand (HNZ) which was established in 2022 to replace 20 district health boards in the expectation it would be more cost-effective and deliver more consistent services.
Austria did the same with their national health insurance, to reduce costs and organisational overhead. It massively increased costs and organisational overhead.
Sometimes with big policy stuff I wonder how much is the effect of the policy vs the effect of the overall societal inertias and momentums there are. Like regardless of that policy, would costs have massively increased (inflation, financialization of healthcare)? Would organizational overhead have increased? Has it decreased in other industries? The policy might still have made it worse, of course.
It's most likely that technical problems are turned into organization departements. The people making the decisions lack the knowledge and tools to solve them efficiently. They gain nothing from solving the problem itself but they gain from growing the organization in size. The solution is obvious while the path to get there can be very convoluted.
Interesting, as a German annoyed by having 100+ insurances I always thought Austria is doing the right thing. Do you have a link for that? I am interested in understanding why the costs increased.
I am only talking about the statutory health insurance providers not the private ones. For the private ones you might be correct. The statutory ones, they have a catalogue of services, which they are required to pay for. While there are some minor differences between the providers, they all offer the same benefits essentially.
Did the Austrian government or any other social western government ever manage do do anything that lead to a decrease in bureaucratic waste? From my PoV, no matter what they do, no matter their marketing messages, the costs always go up and efficiency goes down.
The world runs on Excel. Contrary to popular belief that ERPs and what not is what companies use. Big companies use Excel to verify the ERP data and reports.
There was an article shared here on HN sometime in the middle of last year that broke down how versatile Excel is and how different companies are trying to steal just a niche use of Excel (I remember reading about Airtable doing the automation bit much easier than in Excel, some company doing charts easier than Excel and so on). I wish I could find it again.
"The world runs on Excel" - I think it's true and we should bring AI tooling inside the spreadsheet, not build yet another SaaS silo from where your data will never escape.
Also opens a lot of doors for integrating with private tools or data sources, especially with MCP tooling or such. I published a POC of what I mean at https://supersheets.ai/.
In simple terms: because it is a tool everyone uses and knows to some degree. It has all the bells and whistles to do pretty much what you want with it. And, I guess one overlooked fact: the data is portable, everyone* in the world has Excel.
My view is that if you switch companies, and at the last you used SAP and the new one you have to use Oracle, it is going to be a learning curve with very little added value for you (except the fact that now you know to proprietary systems). With Excel, well, it is going to be the same Excel, just different data.
The fact that it's already installed on almost everyone's computer in any given corporate environment also has a lot to do with why it's so widely used for these types of things I think - noone needs to ask permission to get a new app installed, or jump through hoops to pay for another new piece of software.
as someone who works in the enterprise world as an accountant with some programming experience...
excel is flexible and easy for reporting/analysis for data of varying sizes from small to not ginormous whereas doing anything in an ERP requires an, often, low paid SWE with a thick accent in a different time zone at, comparatively, high cost and big delay
i.e., I can often solve problems in excel in an afternoon that would take months to solve with one of these SWE's
For the same reason Python is so popular. It's not the best at anything (besides being really easy to use), but it's top 5 at all the things that matter. When your use case outgrows Excel, then yeah, invest in a specialized system... when budget allows (which is never).
The same reason we have not been able to evolve beyond a spoon. Once a tool matches a use-case really well, the best you can do is make it worse for the sake of being modern
I've seen several multi billions companies running big processes in Excel. No automation, just put more human workers on the spreadsheets.
I understand the decision though, because investing in a IT system is so friggin expensive, require 10x more people and there is a huge lack of talent to deliver something on time, that works, and actually saves money.
I highly doubt the assessment that it would take 10x more people. If you let a consultancy do it, sure. Any normal medium sized IT company no. The problem with the assessment is invested time, since the invested manhours and failures over all the years of dodgy excel sheets is simply unmeasurable.
The typical problem is that the organisation simply does not have the competence at the correct level to actually order the correct thing from the medium-sized IT company.
In the past, the organisation has probably tried (maybe even several times) to replace the Excel mess with an IT system, only to receive something that costs them a lot of money and solves the wrong problem..
Interesting gap in the market there, someone who can solve that can probably make a lot of money. With the help of AI, of course! /s
I guess the solution is to have an intermediary that can be the in-between, but don't have them be from the IT systems supplier, because they'll bone you (including promising you everything you want without checking with their developers if any of it is possible; with a good 3rd party they'd be able to ask the devs if it's possible and to see a proof-of-concept).
The problem is that is usually costs more than they are willing to spend in a year to get a working system. It's hard to justify spending ~1 million to replace an existing working system. Even if it would pay for itself after a couple of years, business finances don't really work that way.
I'm not sure it's really a matter of initial investment.
Excel allows managers/staff to iterate/change/adapt faster and independently while changing anything about an automated system requires much effort (in communication, in validation, in budget)
This makes me think of a system that just got replaced where I work - multiple hundreds of temporary workers being tracked through their setup and onboarding processes through one massive Excel spreadsheet, saved on a shared Teams group that everyone edited at once. It was held together with prayers, fickle filters, custom views and loose validation rules and I hated it the whole time it was active. We'd frequently have to make sure to remove rows from it and put them in an archive spreadsheet to keep the main one performant enough to function - but that split made it difficult to sift through data when someone had an issue. Not to mention that records routinely went missing or got overlooked.
It's now a SharePoint database - I don't know what I think of that, but at least it was built for the job and has some solid integrations with other systems in it that make life easier.
Not surprised...I am implementing ERP system for 20 years now and Excel is the most popular ERP system :(
Related story when NHS England copied-pasted stuff for COVID stats but most of the data got lost because they are using an old version and there was a 65k line limit...
Perhaps this is telling us that the financial losses and inefficiencies incurred from these supposedly error-prone processes are small enough (relative speaking at least most of the time) that there’s no real consequences to the stakeholders (politicians or bureaucrats in this instance) or the survival of the organisation (this org isn’t for-profit after all). So from the stakeholders’ POV it just doesn’t justify putting the investment upfront into proper software systems. Without greater accountability and real stake in the org from the politicians and bureaucrats, headlines like this will continue to appear in the decades to come.
Headlines like this are more pleasant than headlines about how the $100M project to implement SAP is five years behind schedule, how Deloitte has consumed tens of millions of dollars that could have gone to patient care, or that the new ERP system has lead to a total lack of accountability for patient outcomes.
It's all about risk. Maybe you can run Finance on a spreadsheet and get away with it, and maybe you can survive driving without wearing a seatbelt too.
And once again Felienne Hermans is proven right about the world running on spreadsheets and that we should take them a lot more seriously for that reason alone.
Excel is an incredible swiss army knife that can be used to do a lot, but it has a mean-time-to-catastrophe. You can keep spreadsheet risk (==cost) under control for a long time, which can make it seem like the risk is negligible, but it's a ticking time bomb.
Yes, a benevolent dictator would want to make financial reporting as difficult as possible as they rob the treasury before the "aviator sunglasses" phase.
I worked in a line of business leadership role for a multi-billion dollar organization. The ledger/financial ops stuff for super high-level reporting and payments were done by old mainframe apps. Everything else was done on a series of Excel spreadsheets hosted on a fileserver somewhere. All sorts of complex stuff was done in there. Finance guys are good at finance and frankly, stuff worked.
They eventually transitioned to a modern at the time Peoplesoft system. The spreadsheet system was better in almost every way to the business units. The ERP was better for the financial ops teams. The shortfalls of the various ERP modules meant that the excel work was pushed down to the lines of business -- we probably had exponentially more spreadsheets, made by idiots who had no idea what they were doing. It also worked, but it was slower and certainly alot more expensive. The tools available for things like budgeting were primitive and ineffective - I literally used a P-Card to buy quickbooks and a built my own little finance shop.
Yep, yep. There's yer trouble. Larry found another app to stack on his DB and he knows he's got the finance guys by the balls with vendor lock-in, so it doesn't matter how crap it is.
Fortunately under my BDFL rule he's going to Devil's Island.
> before the "aviator sunglasses" phase
Before? Already there. Got the combination cap too. Been practicing the smile I'll use for the portrait mural that every large building has to have painted on the side.
Sufficiently secure, scalable and reliable for a large organization. Usually implies that vendor can offer SLA for support and security updates, training etc. See for example the difference between RHEL and Fedora. In context of bespoke ERP it means that there are at least 2 tiers of vendors: top tier is responsible for building the integrated solution, underlying tiers are providing individual components for it.
How does something like SAP solve this problem? I would have loved to just drive by but actually let's have a discussion about it. Even Odoo or something more "move fast". How would I sell the product to someone who runs their excel spreadsheets and is price concious so will actually balk at the pricetag to replace the thing that works fine and passes audits.
> replace the thing that works fine and passes audits.
You surely don’t mean Excel, do you? It did not pass the audit.
The main selling point of any ERP is that it may be used to solve the problem in a cost-efficient way. And the problem here is not a single report or aggregation of data sources, it is an enormous list of requirements, implementing which with a manual or semi-automated processes based on Excel would be even more expensive or risky. Those guys in NZ failed even to understand what they actually need (i.e. produce that list of requirements).
Good question! Feel this is just a vague way of referring to formal compliance requirements for software rather than technical functionality supporting large scale operations.
But does anyone use them? Everyone org I've worked for and with - large government departments, publicly traded entities, small PE owned entities use excel.
As others say in this thread, the world runs on Excel.
Sometimes those enterprise tools are used mostly as glorified spreadsheets. I see a lot of BI tools that are really being used to make big spreadsheets.
> The report found “monthly financial reporting usually took 12-15 days to consolidate and five days to analyse.”
> It gets worse: Health Minister Simeon Brown last week delivered a speech in which he said HNZ operates “an estimated 6,000 applications and 100 digital networks” – or roughly one application for every 16 staff members.
> He also said HNZ’s Senior Leadership Team members “have only just begun weekly in-person meetings, and have continued to operate from different offices, despite the majority living in Auckland and the organization being two and a half years old.” Most work in different buildings.
> However the Minister doesn’t have a plan to get off Excel, or fix HNZ’s tech leadership woes.
HNZ's Senior Leadership should be reassigned to clearing fatbergs from sewer mains, and some competent grownups brought in.
And probably similar, further up in the NZ government. Two and a half years is how long it took for the 1940's USA to go from Pearl Harbor to D-Day.
It’s not going well. But keep in mind that running down the public sector and privatising it is their aim. Simeon Brown is a terrible choice for minister and takes some pretty shitty positions for someone in his position.
Also see ‘the financial black hole’ that ACC was supposedly in when John Key came along. It’s the playbook being played.
Don’t look to the US for guidance though, that healthcare system and that political direction is going to be fascinating to look at in two and a half years.
Don't accept anything Simeon Brown says without a healthy dose of skepticism, he's talking shit up for political reasons, like some people in the USA are doing right now.
And if you do choose to accept it as unvarnished truth, welp, that's an ideological choice, not a rational choice.
The current government would dearly like to bring in more private providers to our healthcare systems, so are following the classic neoliberal playbook.
1) Underfund public services
2) Point out how poorly they're performing
3) Bring in private companies to do it "more efficiently"
4) Pay them the money that you refused to pay the public service. And then some more.
5) Private companies make bank
6) Services further worsen.
7) But hey, shareholder value was great.
If you want an example of this, just research the history of our railway system.
Or, you know, the last National government when the Minister of Health directed funding for surgery from the public system to the private system.
And then resigned to become the CEO of a private healthcare company.
Which is why when NZ comes in the top 3 of "least corrupt" countries, I painfully point out that we're in the top 3 of _perceived to be_ least corrupt countries.
Once again, The debacle that was NZRail -> TranzRail -> Toll -> KiwiRail is illustrative.
Toll, an Australian company, ran down our rail network even harder than TranzRail did, and then forced the government to buy it off them at a massive premium by threatening to just shutter our entire rail system.
And we gave fucking knighthoods to the two merchant bankers who advised the government to sell NZ Rail... _and then bought it_ in a consortium with an American company which asset stripped it and then chose not to maintain the railway lines because it provided them a significant tax write-off in it the US on accounts of depreciation or similar.
And of course, Sir Fay and Sir Richwhite now reside in overseas tax havens, although IIRC one of them still owns an entire island here.
Oh, and the NZ government had to pay for the release of one of their children when they were detained driving through Iran for their Instagram account about how everyone is kind.
Ah, neoliberalism.
(As you can tell, I have some strong feelings on this. Because I lived through it. Was it all bad? No. It gave us one of the most efficient agricultural sectors in the world.
Was it mostly bad? Well, inequality surged upwards in conjunction with neoliberal policies. So, yes.)
From what little information there is in the article, it sounds like lack of process was equally to blame as using Excel.
(Not that I would suggest using Excel, personally - I'd go with a small database and a simple web UI but then I'm not a Corporate Consultant looking to charge ££££ per day to faff around for 5 years and then abandon the project.)
There's nothing simple in running accounting in a healthcare system with 16B budget. A small database with simple web UI would have the same problems as Excel spreadsheet. The list of requirements for software automation of such thing would be on four digit number of pages scale. It does require a proper ERP.
>The healthcare system of a country was run in Excel. Didn't you read the title?
Obviously I have read not just the title, but also the article.
>I said the fact that it could be used for that, makes it an impresive achievement for the app
You can try to use a fork to dig a tunnel, but it is hardly an impressive achievement of the fork that it can be used this way, neither you could possibly dig anything big with it. It would be an impressive achievement for Excel, if it actually worked, but the article mentions that there were problems with accounting done this way.
Excel is a powerful tool that can handle massive amounts of data, no doubt. However its feature set is not enough for this kind of tasks. As @zimpenfish said in another comment in this thread, the problem was both in Excel and in processes. Task-appropriate tools help to build good processes around them.
If it creates long term value, is it truly the wrong tool?
Perhaps we should re-evaluate how and when we use things when presented with evidence that others are getting massive value out of tools we typically would have been biased against.
I think so. Using SQLite for an application of that scale would require lots of additional work around caching and queuing at the very least. Much of that could be reduced by using Postgres or some other database designed for that workflow.
Still, if many successful enterprises started SQLite for things of this scale, I should probably reach out to them to see whether there's anything missing from my analysis.
Honestly, I'm surprised that they didn't pay a foreign IT company $100 million to for software less functional than a spreadsheet like our government departments usually do.
Excerpt from the actual review document (Health NZ Financial Management Review, [0]):
Notably, one major issue was through a significant reliance on the use of an Excel file to manage the consolidated financials of the organisation. This spreadsheet was the primary data file used by HNZ to manage its financial performance. It consolidated files from each district into a single spreadsheet, and key reports, such as the monthly finance report, were produced from it. The use of an Excel spreadsheet file to track and report financial performance for a $28bn expenditure organisation raises significant concerns, particularly when other more appropriate systems are present on the IT landscape.
This Excel file is flawed in that:
• Financial information was often 'hard-coded,' making it difficult to trace to the source or have updated data flow through.
• Errors such as incorrectly releasing accruals or double-up releases were not picked up until following periods.
• Changes to prior periods and FTE errors in district financial reporting Excel submissions, would not flow through to consolidated file.
• The spreadsheet can be easy to manipulate information as there is limited tracking to source information where information is not flowing directly from accounting systems.
• It is highly prone to human error, such as accidental typing of a number or omission of a zero.
--
[0] https://www.tewhatuora.govt.nz/assets/Uploads/HNZ-Financial-...
So actually top level planning/reporting is done by everything flowing into one big spreadsheet - as crappy as that is, thats pretty common. Thats not the same as "managed finances with single Excel spreadsheet". Like, they do have other systems.
25 years ago a small Norwegian company in oil industri replaced a big Excel that it were using to manage its inventory with Access database and discovered that they have like extra 200K USD in equipment.
The reason for the saving was enforcement of constrains in various Access tables so a subtle entry error in Excel was discovered.
The transition to DB file was done by a single person in a month who had not worked previously with DB (however he was young and talented programmer I must admit).
And the Access DB in a single file with its UI was quite similar to Excel so guys maintaining it quickly learned it and really liked it.
So it is still puzzles me why such single file DB with nice UI are not more popular? Why it is always Excel?
Someone builds an access database and it works for a bit but then they leave and no-one else knows how it works so they start exporting the data to excel instead. Access is all about restrictions (the form has these fields; the data is structured like so; input will be validated by these rules) ... Restrictions deliver some value but are brittle - its about as hard to maintain as any other custom built app would be.
Access was much easier to get started with than any other similarly capable database product, last time I checked in 2017.
Its easy to get started, but modifying one made by someone else? You've got to really think like a programmer.
as an anecdote, my own mother, who studied french in university and works in education, has built access DBs for places she's worked before. It seems its a very approachable and beginner friendly UI for non-programmers, somehow.
Because Microsoft decided to kill/neglect access and Visual Basic... for reasons?
> why such single file DB with nice UI are not more popular?
Because it needs skills that often aren't immediately available whereas a lot of people have enough Excel experience to lash up even a moderately complex spreadsheet.
At many places there's also some nasty politics around siloing where a database is an IT System so it has to be owned by the centralized IT organization, operated by a DBA team, have a budget, be audited by security, etc. and half of those groups look down on Access and want to architect for job security by picking some Oracle/SAP product where even if you have an unlimited budget you are looking at years of guaranteed delay before you can use it with a significant risk of failure.
In a fair number of places that I've worked the finance and IT departments have a fairly substantial level of enmity which also discourages use of anything which involves them. Meanwhile, everyone still needs to do their jobs and there's a business analyst with decent Excel chops who will have something you can use “temporarily”. A few years pass, then a few more, and now everything lives in Excel as the calcified scar tissue of broken official processes…
> At many places there's also some nasty politics around siloing where a database is an IT System so it has to be owned by the centralized IT organization, operated by a DBA team, have a budget, be audited by security, etc. and half of those groups look down on Access and want to architect for job security by picking some Oracle/SAP product where even if you have an unlimited budget you are looking at years of guaranteed delay before you can use it with a significant risk of failure.
A space seemingly ripe for Line Of Business Applications As A Service: LOBAAAS :-)
TBH, the whole SaaS took off because of the ability for individual departments to just get their resources using nothing but a company credit card.
Excel is really easy to use for smaller data collection and manipulation. There's no input validation, signing and other complicated design decisions needed to start an excel spreadsheet. There's alot of tools out there for easy integration. So its not something you need to switch away from once you have rather static requirements. Billions are just numbers that fit inside a cell just fine until someone complains.
And then once you need actual calling Access lags far behind even MySQL.
FWIW, having a human regularly count the equipment would have also uncovered the error. As well as other interesting reasons that discrepancies arise, such as theft. Always important remember that these systems are tools, not the truth, and have to be reconciled to the truth on a regular basis.
To be fair, you can define value constraints in Excel as well.
> The transition to DB file was done by a single person in a month
If Deloitte is going to do this it will take 3 years, take about 20 people and cost tens of millions.
>So actually top level planning/reporting is done by everything flowing into one big spreadsheet - as crappy as that is, thats pretty common.
Yeah, everywhere I've worked that's common, even when the underlying data is managed much more successfully, the top folks really want everything on a spreadsheet.
Yea exactly my take. This seems like exactly what Excel is good at doing.
You are mistaking "being commonly used for this task" with "good at doing"
Replacing this excel sheet with a 'proper' system would cost between $5million and $20million probably, depending on what sort of consultancy delivered it
And would be inflexible, and maybe still wrong, and not necessarily more transparent... So it may mean several extra million dollars per year in ongoing maintenance if things turn out typically...
Excel has many issues, but the cost of replacing it is surprisingly high.
>And would be inflexible, and maybe still wrong, and not necessarily more transparent...
I'm going through this right now. Not anywhere in this scale, but just in terms of deciding whether to move an order-tracking/accounting system I've built in Excel to a "real" database.
The database I am considering (Panorama X for Mac) is quite inexpensive, I've heard very good things about it, and has a spreadsheet-like UI. However, I've used Excel enough to know that I haven't tapped more than a small fraction of its ability, especially things like Power Query. As much as I loathe VBA, what if the cost of moving to a "real" database isn't the up-front cost, but the longer-term inflexibility of Panorama (and, pretty much, anything else in my price range) compared to the beast that is Excel?
Probably why the consultancy who produced the report referenced here was delighted to point it out.
Anything commonly used for a task it’s good enough for that task even if “much better” options exist.
Nail guns have a lot of downsides vs using screws, but being much faster and cheap offsets quite a lot.
Just bad enough
Excel isn't 'good' at anything. It is passable at a great many things.
Quantity has a quality all its own.
It really does.
Just the massive, massive ecosystem of online tutorials and answers and YouTube videos for every conceivable question [1] is incomparable.
[1] Except the one you want the answer to at a given moment, amirite fellas?
>[1] Except the one you want the answer to at a given moment, amirite fellas?
Everytime I relearn how to do a vlookup, none of the examples online do what I want. It's like they are trying to do something completely opposite of what I want. It's super weird, because what I want to do is always a pretty common thing to need done.
Sounds like a certain LLM I've read about
> This spreadsheet was the primary data file
>> - as crappy as that is, thats pretty common.
No. This is not a common practice. An Accounting / ERP system with full audit trails and data entry compliance is. Exporting data from such a system into Excel for analysis is. But Excel as the [re-reads] Primary data file is not a practice at all.
I've worked on dozens of Excel to ERP migrations. Excel is way more common as the primary data source including financial data. If you only look at financial data by volume (and not count) maybe ERP win, but I wouldn't be sure.
> and was used for “consolidation, journals, business-critical reporting, and analysis.”
Not sure what 'journals' is supposed to mean here but “consolidation, business-critical reporting, and analysis.” does not mean operations day to day use, it means high level management reporting
My position is that the article (or the consultants who are trying to sell the ERP) are exaggerating the role of the spreadsheet.
In accounting, a "journal" is a chronological record of financial transactions.
"primary", I hope as a commenter out here you understand what that means.
If the Excel file is just a snapshot of the financial situation and it doesn't feed back into the main system, I find it harder to see the issue. Essentially it sounds like a manager takes the Excel file, makes some small changes, writes some equations, produces tables/graphs and then writes a report.
Maybe there is an error in the reporting, but it can always be corrected. The idea isn't to explain where $16.129626356 bn dollars went, it's to explain where ~$16 bn dollars went.
I can think of many processes just off the top of my head where commercial software outputs something like a CSV file, I add my data to it, then generate a report. I don't think the process is insane at all. We catch errors by predicting the data to be entered and flagging if off by 10%, and then also looking at the distribution of the data entered (it should look like a skewed bell curve). We then have a stand-up meeting where we review the outputs and then have a process for later correction. Of course it's possible for errors to creep in, but the errors are very unlikely and have somewhat limited scope.
My point is that the situation in NZ seems entirely resolvable, and they shouldn't just throw it out if it was somewhat working. The whole "Excel" thing seems like a nothing-burger.
If you can't explain where NZD129m of spending went, you screwed up big time.
There's a difference between "can't explain" and "isn't immediately visible on a high-ish level report"
If you spend a billion and a couple years redoing the generally working system to (maybe!) increase the number of significant digits on a dashboard for no good reason, you may have screwed up even more.
These same bullet points and half the first paragraph are quoted in the OP article fwiw.
I think the important part that's not clearly portrayed in the OP article is that the "one Excel file" is actually not a single file used across the whole organisation, but rather a sort of master overview file that "consolidated files from each district into a single spreadsheet".
I don't think that fact is made very clear in the OP article.
At page 42. Not a random number.
Read this as Deloitte trying to sell some $1b worth of consulting to replace a spreadsheet.
Some of those defects seem as if they could be used to improve spreadsheet software more than they mean that spreadsheet software is intrinsically inappropriate.
The amount of issues I have seen in the last ~20 years caused by "off by one" type of errors in Excel is insane.
Few examples:
- incorrect schedule for the electricity grid for an entire country
- incorrect assessment of an airport use for an airline (causing few millions USD loss in revenue)
- incorrect financial position assessment for a mine (resulting incorrect deciosion to optimize the wrong business process, not sure about how much they lost)
Making illegal states unrepresentable is a concept that benefits programming langues and business processes alike.
You’re only looking at one side of it. Consider how much value that excel adds, and how many use cases it enables. It’s incalculable. It’s always easy to poke at excel’s issues, but errors are almost always a skill issue.
From a financial perspective, there are many ways you could enforce checks to ensure the model is balanced - it just takes time. Data entry can be an issue, but you can automate that too. The deloitte report is saying the health department should benefit from adopting a gigantic erp system but you could get 90% of the benefit by employing a couple people that really know what they’re doing.
You could say that excel should allow those things to happen but the flexibility is precisely what makes it so valuable.
The people writing the reports are consultants. Consultants recommend things that benefit consultants. In their case, a multi year process and tens of millions of dollars trying to install ERP software is more a windfall for the vendors and not the companies.
Absolutely, kudos to them for following KISS, despite the legions of people who have probably told them they’re doing it wrong.
Yep. I was going to say the same thing. Installing some $10M+ software costs the company more than just the install and ongoing licensing costs. You have to have an entire new IT team to manage it and staff to understand and so on. There are probably just as many chances of having a big issue.
That assumes that Better Excel would somehow not look like Excel and would lose its benefits of being easy. My assumption is that Better Excel will still look like Excel, in the same way that Better iOS will also look like iOS.
> but errors are almost always a skill issue
Sure but to compare it to tech, this is why we build tooling like linters and typed languages so that we don’t lose millions of dollars because we needed the average developer to implement a feature. I imagine in heath care it’s even more skewed because a SW eng can see the compiler output and make changes but the person entering the data isn’t (for example) the nurse that’s using the needles to be accounted for
There are indeed tools like this available for Excel. And there are also whole practices within the big accountancy firms, and independent firms which focus on providing "audit" of Excel files like this, which are used commonly where there are financial firms involved in a project with big money at stake (lenders, financial investors, etc). But in my experience these tools and services aren't often used for the spreadsheets which are used for day to day corporate-type management (I would count government as part of that).
Say nothing of many crisis were avoided because somebody was able to pull up a local excel file despite a central service being down or go through their historical files and sniff out a discrepancy. Workflows like that are often impossible or require coordinating with a vendor in another time zone or a bunch of slow cross team communication and ticket submissions with the sort of minimum effort "we'll run a DB in the cloud and slap a JS front end built with a bunch of questionable but flashy libraries and ignore all the edge cases because I'll be working somewhere else in 2yr" solutions that the median software developer hive mind will implement if left to its own devices.
Do you have evidence that _any_ other software would not be prone to this kind of errors, taking into account the fact that at the scale of dozens of billions of dollars, the numbers you mentioned seem like rounding errors (except for the first one which would deserve a reference)?
>The amount of issues I have seen in the last ~20 years caused by "off by one" type of errors in Excel is insane.
Generally those aren't a fault of excel, but a fault of someone doing something dumb in excel. People do dumb stuff in every problem and it's nearly impossible to prevent it.
Don't forget Reinhart-Rogoff, which ties into the other thread on academic fraud/nonreproduction, and arguably did huge amounts of economic damage: https://www.bbc.co.uk/news/magazine-22223190
IMO unit tests are roughly the software equivalent of double-entry bookkeeping.
This is a very good analogy. I will steal it.
Imagine how many errors occurred before we had digital spreadsheet technology.
That said, I’ve always thought there was a product that sits somewhere in between Excel and full blown custom software that provides some of the controls we need while still being and build able by someone with low/average technical skills
FileMaker seems to cover this (admittedly my interactions with it was in high school, which was last millennium), it's a database app but there's a lot of "make your own UI" parts in it, so you can create custom UIs (and wizards) for your use cases.
I suppose MS Access offers this too, although FM feels more user-friendly, MS Access felt like Internet Explorer 4, where an error dialog would pop-up for every little JavaScript error (disclaimer: this opinion is from 25 years ago).
Filemaker is great. We created a monster at our University to keep track of Research Grants and it worked for almost 20 years flawlessly.
Untill, a consultant replaced it with an 'enterprise' solution that cost 10000x more to run and maintain.
Sometimes, well designed systems, however simple they might be, would be all that is required.
No you're right on the money in my opinion. I have very fond memories of spinning up apps in MS Access that were quick and dirty but extremely powerful, and able to be maintained/modified by folks that were not professionally trained software engineers. I think there's a missing area in the market for something exactly like that again.
The hive mind really hated (hates?) MS Access and I don't know why.
I built a mini ERP for my father's company when I was 16. Started out as an excel spreadsheet, but then added inventory, accounting, and printing contracts and reports.
He kept using it until retirement and was very happy with it. I could learn & apply SQL. Win-win.
I've done that over the last several years, building an order-tracking/accounting system for myself in Excel. (Inventory and fulfillment is handled by commercial software.) I'm mulling over whether to move to a "real" database.
I'm on a Mac so can't move to Access. I've thought about FileMaker, but am considering Panorama X because unlike FileMaker it reportedly allows undoing almost anything, while FileMaker is like the typical database in a record commit not being undoable.
(Yes, I know that not allowing such is good practice, which is why I am not a database admin.)
I've heard good things about Panorama X, and it has a spreadsheet-like UI. However, I've used Excel enough to know that I haven't tapped more than a small fraction of its ability, especially things like Power Query. As much as I loathe VBA, what if the cost of moving to a "real" database isn't the up-front cost or conversion time, but the longer-term inflexibility of Panorama (and, pretty much, anything else in my price range) compared to the beast that is Excel?
I've not used it, but I thought Airtable occupied this spot. It seems to have a few open-source clones if you want to run it locally. I don't know if they are any good.
Microsoft also has Power BI which occupies this "database which looks like a spreadsheet" space. But it's not well known.
> The spreadsheet-using agency is Health New Zealand (HNZ) which was established in 2022 to replace 20 district health boards in the expectation it would be more cost-effective and deliver more consistent services.
Austria did the same with their national health insurance, to reduce costs and organisational overhead. It massively increased costs and organisational overhead.
Sometimes with big policy stuff I wonder how much is the effect of the policy vs the effect of the overall societal inertias and momentums there are. Like regardless of that policy, would costs have massively increased (inflation, financialization of healthcare)? Would organizational overhead have increased? Has it decreased in other industries? The policy might still have made it worse, of course.
It's most likely that technical problems are turned into organization departements. The people making the decisions lack the knowledge and tools to solve them efficiently. They gain nothing from solving the problem itself but they gain from growing the organization in size. The solution is obvious while the path to get there can be very convoluted.
Interesting, as a German annoyed by having 100+ insurances I always thought Austria is doing the right thing. Do you have a link for that? I am interested in understanding why the costs increased.
https://www.derstandard.at/story/2000137202435/krankenversic...
tl;dr: they promised savings of one billion Euros but it instead cost over 200 million.
Ok, the costs saving was promised by the populists Kurz and Strache. No further questions. Thanks for the link, tho!
I am not saying Germany should merge all insurances into one, but I still think 100+ is one order of magnitude too much.
Doesn't 100+ insurances mean more competition leading to better prices? OR so I'm told capitalism works.
I am only talking about the statutory health insurance providers not the private ones. For the private ones you might be correct. The statutory ones, they have a catalogue of services, which they are required to pay for. While there are some minor differences between the providers, they all offer the same benefits essentially.
>Doesn't 100+ insurances mean more competition leading to better prices? OR so I'm told capitalism works.
I'm sure this was /s but capitalism doesn't work for things like insurance.
Did the Austrian government or any other social western government ever manage do do anything that lead to a decrease in bureaucratic waste? From my PoV, no matter what they do, no matter their marketing messages, the costs always go up and efficiency goes down.
Did any government, western or not? Without making other things significantly worse I mean.
Maybe Denmark?
I had a minor role in that project... A total shit show.
The world runs on Excel. Contrary to popular belief that ERPs and what not is what companies use. Big companies use Excel to verify the ERP data and reports.
There was an article shared here on HN sometime in the middle of last year that broke down how versatile Excel is and how different companies are trying to steal just a niche use of Excel (I remember reading about Airtable doing the automation bit much easier than in Excel, some company doing charts easier than Excel and so on). I wish I could find it again.
"The world runs on Excel" - I think it's true and we should bring AI tooling inside the spreadsheet, not build yet another SaaS silo from where your data will never escape.
Also opens a lot of doors for integrating with private tools or data sources, especially with MCP tooling or such. I published a POC of what I mean at https://supersheets.ai/.
I think you're looking for Excel Never Dies on Packy McCormick's Not Boring newsletter [0]. Featured previously on HN [1].
[0]: https://www.notboring.co/p/excel-never-dies [1]: https://news.ycombinator.com/item?id=32346288
Why do you think have we not been able to evolve beyond this in the enterprise world?
In simple terms: because it is a tool everyone uses and knows to some degree. It has all the bells and whistles to do pretty much what you want with it. And, I guess one overlooked fact: the data is portable, everyone* in the world has Excel.
My view is that if you switch companies, and at the last you used SAP and the new one you have to use Oracle, it is going to be a learning curve with very little added value for you (except the fact that now you know to proprietary systems). With Excel, well, it is going to be the same Excel, just different data.
The fact that it's already installed on almost everyone's computer in any given corporate environment also has a lot to do with why it's so widely used for these types of things I think - noone needs to ask permission to get a new app installed, or jump through hoops to pay for another new piece of software.
as someone who works in the enterprise world as an accountant with some programming experience...
excel is flexible and easy for reporting/analysis for data of varying sizes from small to not ginormous whereas doing anything in an ERP requires an, often, low paid SWE with a thick accent in a different time zone at, comparatively, high cost and big delay
i.e., I can often solve problems in excel in an afternoon that would take months to solve with one of these SWE's
For the same reason Python is so popular. It's not the best at anything (besides being really easy to use), but it's top 5 at all the things that matter. When your use case outgrows Excel, then yeah, invest in a specialized system... when budget allows (which is never).
The same reason we have not been able to evolve beyond a spoon. Once a tool matches a use-case really well, the best you can do is make it worse for the sake of being modern
I've seen several multi billions companies running big processes in Excel. No automation, just put more human workers on the spreadsheets.
I understand the decision though, because investing in a IT system is so friggin expensive, require 10x more people and there is a huge lack of talent to deliver something on time, that works, and actually saves money.
I highly doubt the assessment that it would take 10x more people. If you let a consultancy do it, sure. Any normal medium sized IT company no. The problem with the assessment is invested time, since the invested manhours and failures over all the years of dodgy excel sheets is simply unmeasurable.
The typical problem is that the organisation simply does not have the competence at the correct level to actually order the correct thing from the medium-sized IT company.
In the past, the organisation has probably tried (maybe even several times) to replace the Excel mess with an IT system, only to receive something that costs them a lot of money and solves the wrong problem..
I have seen projects being implemented in SAP, by 20 people, that a single junior software developer could do better solo in the same time frame.
Interesting gap in the market there, someone who can solve that can probably make a lot of money. With the help of AI, of course! /s
I guess the solution is to have an intermediary that can be the in-between, but don't have them be from the IT systems supplier, because they'll bone you (including promising you everything you want without checking with their developers if any of it is possible; with a good 3rd party they'd be able to ask the devs if it's possible and to see a proof-of-concept).
The problem is that is usually costs more than they are willing to spend in a year to get a working system. It's hard to justify spending ~1 million to replace an existing working system. Even if it would pay for itself after a couple of years, business finances don't really work that way.
I'm not sure it's really a matter of initial investment.
Excel allows managers/staff to iterate/change/adapt faster and independently while changing anything about an automated system requires much effort (in communication, in validation, in budget)
> deliver something on time
Or ever.
This makes me think of a system that just got replaced where I work - multiple hundreds of temporary workers being tracked through their setup and onboarding processes through one massive Excel spreadsheet, saved on a shared Teams group that everyone edited at once. It was held together with prayers, fickle filters, custom views and loose validation rules and I hated it the whole time it was active. We'd frequently have to make sure to remove rows from it and put them in an archive spreadsheet to keep the main one performant enough to function - but that split made it difficult to sift through data when someone had an issue. Not to mention that records routinely went missing or got overlooked.
It's now a SharePoint database - I don't know what I think of that, but at least it was built for the job and has some solid integrations with other systems in it that make life easier.
This makes me feel better about a current situation, briefly.
Not surprised...I am implementing ERP system for 20 years now and Excel is the most popular ERP system :(
Related story when NHS England copied-pasted stuff for COVID stats but most of the data got lost because they are using an old version and there was a 65k line limit...
https://www.bbc.co.uk/news/technology-54423988
What a truly powerful tool!
Perhaps this is telling us that the financial losses and inefficiencies incurred from these supposedly error-prone processes are small enough (relative speaking at least most of the time) that there’s no real consequences to the stakeholders (politicians or bureaucrats in this instance) or the survival of the organisation (this org isn’t for-profit after all). So from the stakeholders’ POV it just doesn’t justify putting the investment upfront into proper software systems. Without greater accountability and real stake in the org from the politicians and bureaucrats, headlines like this will continue to appear in the decades to come.
Headlines like this are more pleasant than headlines about how the $100M project to implement SAP is five years behind schedule, how Deloitte has consumed tens of millions of dollars that could have gone to patient care, or that the new ERP system has lead to a total lack of accountability for patient outcomes.
It's all about risk. Maybe you can run Finance on a spreadsheet and get away with it, and maybe you can survive driving without wearing a seatbelt too.
Perhaps it means that the reorganization was unnecessary, and badly executed. Things seemed to go well before.
And once again Felienne Hermans is proven right about the world running on spreadsheets and that we should take them a lot more seriously for that reason alone.
[0] https://www.youtube.com/watch?v=0yKf8TrLUOw
Excel is an incredible swiss army knife that can be used to do a lot, but it has a mean-time-to-catastrophe. You can keep spreadsheet risk (==cost) under control for a long time, which can make it seem like the risk is negligible, but it's a ticking time bomb.
If we simply implemented my benevolent dictatorship for life, we'd have a system under which installing Microsoft Excel became a capital crime.
Yes, a benevolent dictator would want to make financial reporting as difficult as possible as they rob the treasury before the "aviator sunglasses" phase.
I worked in a line of business leadership role for a multi-billion dollar organization. The ledger/financial ops stuff for super high-level reporting and payments were done by old mainframe apps. Everything else was done on a series of Excel spreadsheets hosted on a fileserver somewhere. All sorts of complex stuff was done in there. Finance guys are good at finance and frankly, stuff worked.
They eventually transitioned to a modern at the time Peoplesoft system. The spreadsheet system was better in almost every way to the business units. The ERP was better for the financial ops teams. The shortfalls of the various ERP modules meant that the excel work was pushed down to the lines of business -- we probably had exponentially more spreadsheets, made by idiots who had no idea what they were doing. It also worked, but it was slower and certainly alot more expensive. The tools available for things like budgeting were primitive and ineffective - I literally used a P-Card to buy quickbooks and a built my own little finance shop.
> Peoplesoft
Yep, yep. There's yer trouble. Larry found another app to stack on his DB and he knows he's got the finance guys by the balls with vendor lock-in, so it doesn't matter how crap it is.
Fortunately under my BDFL rule he's going to Devil's Island.
> before the "aviator sunglasses" phase
Before? Already there. Got the combination cap too. Been practicing the smile I'll use for the portrait mural that every large building has to have painted on the side.
Computers with Microsoft Excel already installed on them before your edict would skyrocket in value. :)
https://metacpan.org/search?p=1&q=excel&size=500
Just out of curiosity, what are some professional tools to handle finance information of that magnitude?
Bespoke ERPs implemented by external contractors on top of enterprise-grade database/cloud etc.
> enterprise-grade
What is this?
Sufficiently secure, scalable and reliable for a large organization. Usually implies that vendor can offer SLA for support and security updates, training etc. See for example the difference between RHEL and Fedora. In context of bespoke ERP it means that there are at least 2 tiers of vendors: top tier is responsible for building the integrated solution, underlying tiers are providing individual components for it.
I was looking for the \s in there...
How does something like SAP solve this problem? I would have loved to just drive by but actually let's have a discussion about it. Even Odoo or something more "move fast". How would I sell the product to someone who runs their excel spreadsheets and is price concious so will actually balk at the pricetag to replace the thing that works fine and passes audits.
> replace the thing that works fine and passes audits.
You surely don’t mean Excel, do you? It did not pass the audit.
The main selling point of any ERP is that it may be used to solve the problem in a cost-efficient way. And the problem here is not a single report or aggregation of data sources, it is an enormous list of requirements, implementing which with a manual or semi-automated processes based on Excel would be even more expensive or risky. Those guys in NZ failed even to understand what they actually need (i.e. produce that list of requirements).
Good question! Feel this is just a vague way of referring to formal compliance requirements for software rather than technical functionality supporting large scale operations.
Unbelievable. They should at least manage the excel in git.
I generally try to have staff time stamp their files with date/time and mark them read only - I suppose that's easier than git though not as good...
They did not want to go with "premature abstraction"
This is hilarious.
Is there another budget tool other than excel? It’s the only one I’ve seen working in forecasting and budgeting in private and public sector
There are many, like mosaic.tech or causal.app
But does anyone use them? Everyone org I've worked for and with - large government departments, publicly traded entities, small PE owned entities use excel.
As others say in this thread, the world runs on Excel.
Sounds like the missing technology here is double entry accounting, not the persistence layer.
Blows my mind that these things are still happening despite many large scale ERP implementations being the norm?
Probably better than people with an "enterprise" tools did, too.
Sometimes those enterprise tools are used mostly as glorified spreadsheets. I see a lot of BI tools that are really being used to make big spreadsheets.
The first feature I've ever seen anybody go looking for in any of them is the "export to Excel" (or CSV) function.
Interesting fact: Deloitte reports the use of the Excel file at page 42.
Where all of the answers go.
Ah Excel, the second-best tool for everything.
1. Buy a state of the art supercomputer.
2. name it 'prod'
3. Move ALL infra onto it
4. ????
Hopefully better than ol’ SBF did
Excel runs the world, literally.
> The report found “monthly financial reporting usually took 12-15 days to consolidate and five days to analyse.”
> It gets worse: Health Minister Simeon Brown last week delivered a speech in which he said HNZ operates “an estimated 6,000 applications and 100 digital networks” – or roughly one application for every 16 staff members.
> He also said HNZ’s Senior Leadership Team members “have only just begun weekly in-person meetings, and have continued to operate from different offices, despite the majority living in Auckland and the organization being two and a half years old.” Most work in different buildings.
> However the Minister doesn’t have a plan to get off Excel, or fix HNZ’s tech leadership woes.
HNZ's Senior Leadership should be reassigned to clearing fatbergs from sewer mains, and some competent grownups brought in.
And probably similar, further up in the NZ government. Two and a half years is how long it took for the 1940's USA to go from Pearl Harbor to D-Day.
It’s not going well. But keep in mind that running down the public sector and privatising it is their aim. Simeon Brown is a terrible choice for minister and takes some pretty shitty positions for someone in his position.
Also see ‘the financial black hole’ that ACC was supposedly in when John Key came along. It’s the playbook being played.
Don’t look to the US for guidance though, that healthcare system and that political direction is going to be fascinating to look at in two and a half years.
Good intentions without a good plan and leadership to execute? Or invent bureaucratic incompetence?
Don't accept anything Simeon Brown says without a healthy dose of skepticism, he's talking shit up for political reasons, like some people in the USA are doing right now.
And if you do choose to accept it as unvarnished truth, welp, that's an ideological choice, not a rational choice.
The current government would dearly like to bring in more private providers to our healthcare systems, so are following the classic neoliberal playbook.
1) Underfund public services
2) Point out how poorly they're performing
3) Bring in private companies to do it "more efficiently"
4) Pay them the money that you refused to pay the public service. And then some more.
5) Private companies make bank
6) Services further worsen.
7) But hey, shareholder value was great.
If you want an example of this, just research the history of our railway system.
Or, you know, the last National government when the Minister of Health directed funding for surgery from the public system to the private system.
And then resigned to become the CEO of a private healthcare company.
Which is why when NZ comes in the top 3 of "least corrupt" countries, I painfully point out that we're in the top 3 of _perceived to be_ least corrupt countries.
Once again, The debacle that was NZRail -> TranzRail -> Toll -> KiwiRail is illustrative.
Toll, an Australian company, ran down our rail network even harder than TranzRail did, and then forced the government to buy it off them at a massive premium by threatening to just shutter our entire rail system.
And we gave fucking knighthoods to the two merchant bankers who advised the government to sell NZ Rail... _and then bought it_ in a consortium with an American company which asset stripped it and then chose not to maintain the railway lines because it provided them a significant tax write-off in it the US on accounts of depreciation or similar.
And of course, Sir Fay and Sir Richwhite now reside in overseas tax havens, although IIRC one of them still owns an entire island here.
Oh, and the NZ government had to pay for the release of one of their children when they were detained driving through Iran for their Instagram account about how everyone is kind.
Ah, neoliberalism.
(As you can tell, I have some strong feelings on this. Because I lived through it. Was it all bad? No. It gave us one of the most efficient agricultural sectors in the world.
Was it mostly bad? Well, inequality surged upwards in conjunction with neoliberal policies. So, yes.)
laughs in hedge fund manager voice
Plot twist: this actually serves as more positive advertising for Microsoft Excel.
How so? Apparently, the conclusion is that Excel is the wrong tool for such tasks, so it should be used less, not more.
> Excel is the wrong tool for such tasks
From what little information there is in the article, it sounds like lack of process was equally to blame as using Excel.
(Not that I would suggest using Excel, personally - I'd go with a small database and a simple web UI but then I'm not a Corporate Consultant looking to charge ££££ per day to faff around for 5 years and then abandon the project.)
>I'd go with a small database and a simple web UI
There's nothing simple in running accounting in a healthcare system with 16B budget. A small database with simple web UI would have the same problems as Excel spreadsheet. The list of requirements for software automation of such thing would be on four digit number of pages scale. It does require a proper ERP.
>How so?
The healthcare system of a country was run in Excel. Didn't you read the title?
> so it should be used less, not more
I never said it should be used more, I said the fact that it could be used for that, makes it an impresive achievement for the app.
>The healthcare system of a country was run in Excel. Didn't you read the title?
Obviously I have read not just the title, but also the article.
>I said the fact that it could be used for that, makes it an impresive achievement for the app
You can try to use a fork to dig a tunnel, but it is hardly an impressive achievement of the fork that it can be used this way, neither you could possibly dig anything big with it. It would be an impressive achievement for Excel, if it actually worked, but the article mentions that there were problems with accounting done this way.
Excel is a powerful tool that can handle massive amounts of data, no doubt. However its feature set is not enough for this kind of tasks. As @zimpenfish said in another comment in this thread, the problem was both in Excel and in processes. Task-appropriate tools help to build good processes around them.
I suppose it's kind of like a billion DAU app using a single SQLite file for their DB.
Certainly the wrong tool for the job, but nevertheless impressive for both the technology and the operators if it somehow works.
If it creates long term value, is it truly the wrong tool?
Perhaps we should re-evaluate how and when we use things when presented with evidence that others are getting massive value out of tools we typically would have been biased against.
I think so. Using SQLite for an application of that scale would require lots of additional work around caching and queuing at the very least. Much of that could be reduced by using Postgres or some other database designed for that workflow.
Still, if many successful enterprises started SQLite for things of this scale, I should probably reach out to them to see whether there's anything missing from my analysis.
[flagged]
[flagged]
Honestly, I'm surprised that they didn't pay a foreign IT company $100 million to for software less functional than a spreadsheet like our government departments usually do.
Don’t worry, we have definitely also done that.
This article proves that a) Excel is not the right tool for everything b) people will (over)use Excel even without being advertised to.