Open Source Software and "the Church of England's problem"

No, this isn't a religious post.

I was listening to an interview regarding marketing challenges for big brands trying to reinvent themselves. The context was large, older brands that people don't want to go away but whose products they aren't going to buy (any more).

A comparison was made with the Church of England. (It was a very English discussion.) The speakers  said:

• They wanted it to keep existing.
• They weren't going to go.
• And, they certainly weren't going to pay for it.

I immediately saw a comparison with Open Source Software:

• People want it to exist (and be maintained/updated)
• (Most) people are unlikely to participate. (Beyond using it.)
• And, they certainly weren't going to pay for it.

The challenges facing the CofE and OSS have similarities, but I think the solutions each will seek are very different.

That's all, except to say that the impact on anything is highly likely to be the same when people want something but aren't willing to pay or participate.

Even more testing nonsense with generated tests

So, following on from the last post I wrote.

I wrote that post just before I checked in the code.

As a final check before committing the code, I did some manual checks.

I found a[n obvious] scenario that didn't work.

I thought I'd created a test case for it. Turns out I hadn't. So, I added one and it failed. Then I added an extra clause to the function and made sure all tests pass.

I wonder if the Copilot generated tests cover this scenario?

It turns out they do and they don't.

There is a generated test with a name that implies it covers the scenario, but the actual contents of the test method do not do what the name implies.

It's not enough to have the test, it's also necessary that it's valid and correct.

Review generated tests more thoroughly than generated code.
If the test is wrong it leads to code that is wrong.

This also highlights the need to have other people review code (and test plans).

I know I'm not perfect and I make mistakes and omissions.
That's part of the reason for code reviews: to have an extra pair of eyes double check things. (It's just a shame this was all on a personal project.)

This experience really has reinforced my belief that test cases should be written before code.

The future of software development is testing

Or, to be more precise, the future of work for a lot of people doing "software development" depends on the need to test code.

AI is great. You can use it to write code without having to know how to program.

Except, the real value of a good software developer is knowing what code to write.

Yes, you may be able to eventually get AI to generate code that initially appears to produce what you want, but how can you determine if it's good, or efficient, or correctly covers all scenarios you need?

It's not just a case of having code that seems to work, you also (potentially among other things) need to know if the code is correct. 

AI tools can be useful to software developers by saving time generating code more quickly than the developer could have written it themselves. This is true for almost all developers. Even me.

Today, I needed to write a function to see if two numeric ranges overlapped.

It's quite a trivial thing. There are lots of examples of this type of method in existence. I could easily write such a function, but I decided to let Copilot create it for me.

The code it produced looked fine:

public bool Intersects(int start, int length)
{
    return (start >= this.Start && start < this.Start + this.Length) ||
        (start + length >= this.Start && start + length < this.Start + this.Length);
}

All it needed was some boundary checks to return false if either of the inputs was negative.

That should be good enough, I thought.

But, how can I be sure?

Does it matter if it's not?

I know this is just the type of code that: it is easy to have a mistake in; such mistakes are hard to spot; and any mistake is likely to produce annoying bugs.

So, I wanted to make sure the function does all that I need. That means writing some coded tests.

I also know that Copilot can generate tests for a piece of code.

So, I decided on a challenge: Could can I write better or a more complete set of tests?

I went first and I came up with 15 test cases. 

One of them failed.

But, it was easy to spot the issue with the code, make the change, and rerun the tests to see them all pass.

Then I let Copilot have a go. It came up with 5 test cases.

They all passed. First with the unmodified function.

Then, with the modified function, the tests all still passed.

Being able to change the logic within the function and not see any change in test results is a clear sign that there is insufficient test coverage.

Not that this is about the number of tests. This is about the quality of tests. 

When it comes to testing, quality is significantly more important than quantity.

The question should not be "Are there tests?" or "How many tests are there?"

The question you need to ask is "Do these tests give confidence that everything works correctly?"

How do you ensure that AI generated code is 100% what you want and intend? You have to be incredibly precise and consider all scenarios. 

Those scenarios make good test cases.

If you're working with code, whether it's written by a person or AI, you need to test it thoroughly. And the challenging part of thorough testing, and the bit you can't (yet?) totally outsource to AI is determining all the scenarios (test cases) to consider and account for.

What was the case that failed and how did the code need to be changed?

I needed the code to work with input of a length of zero. If this was at the end of the existing range, I need to consider this as intersecting, but the initially generated code considered it not to be.

The fix was to change the final "less than" to be a "less than or equals" and then it did what I needed.

Having now written this up, I also realize there's a potential for an overflow with internal sums that are greater than Int.Max, but as the reality of this code is that it will only be used with maximum input values of a few thousand this shouldn't be an issue...for now! ;)

This wasn't the end of the story: https://www.mrlacey.com/2024/11/even-more-testing-nonsense-with.html

Open a web page when a project can't be built

When writing the Uno book, we wanted a way to have the project fail with a useful message if a required dependency hadn't also been set up correctly.

Below is what I came up with.

<Target Name="CheckForSfChart">
    <Exec Condition="!Exists('..\..\..\syncfusion-Uno.SfChart\Src\Syncfusion.SfChart.Uno.csproj')" Command="cmd.exe /c start https:////github.com/PacktPublishing/Creating-cross-platform-applications-with-Uno/tree/main/Chapter6/readme.md" />
    <Error Condition="!Exists('..\..\..\syncfusion-Uno.SfChart\Src\Syncfusion.SfChart.Uno.csproj')" Text="The SfChart project is also needed. See https://github.com/PacktPublishing/Creating-cross-platform-applications-with-Uno/tree/main/Chapter6/readme.md" />
  </Target>

See the real thing in GitHub.

If the required other project is missing, it:

• Opens a webpage with more details.
• Outputs a message with details in the build output.

Points of note:

• Yes, msbuild will fail to compile the solution with missing project anyway. This is just additional info to help in debugging.
• Repeated slashes in the url ("https:////" instead of "https://") due to them assumed to be being escaped
• Because this call cmd.exe, it will only work on Windows.
• Not a perfect solution but better than nothing.

Documenting this now before I forget.

The lack of a visual designer is actually offensive

One of the biggest complaints and requests relates to a lack of a "visual designer" for creating UIs with XAML for WinUI3 and .NET MAUI applications.

For a long time, Microsoft had no plans to build one. Although that may be changing...

There are other, related tools that they are working on but are not the designer people are asking for.

I've even heard it said (by people who are employed by Microsoft) that "these tools are better than a designer so you don't need one".

That's what I find offensive.

The claim that MS know what people want better than they know themselves. And, the implication that people are wrong about what they say they want. More importantly (and concerning) is that I don't think MS are listening to the "real" request. 

Yes, the Hot and Live tools are great for working with existing code when the app is running, but there's also a need to help with a completely blank app (or page, or window...)

There needs to be a better way to help people get started and give them confidence that what is in the source is correct before they start debugging (or even compiling?) the code.

I have worked with people creating drag-and-drop-related tools for XAML UIs, so I know what's possible and how hard it is. That's also why I'm not trying to build one myself.

I think there's another way. It's not necessarily better at first look, but it's certainly very helpful, powerful, and able to offer more productivity than arguing about why something that isn't what people are asking for isn't what they want or need.