The factor I discover most baffling in regards to the programming assessments I have been operating is that instruments primarily based on the identical massive language mannequin are likely to carry out fairly otherwise.
For instance, ChatGPT, Perplexity, and GitHub Copilot are all primarily based on the GPT-4 mannequin from OpenAI. However, as I am going to present you under, whereas ChatGPT and Perplexity’s professional plans carried out excellently, GitHub Copilot failed as typically because it succeeded.
I examined GitHub Copilot embedded inside a VS Code occasion. I am going to clarify the best way to set that up and use GitHub Copilot in an upcoming step-by-step article. However first, let’s run via the assessments.
If you wish to understand how I check and the prompts for every particular person check, be at liberty to learn how I check an AI chatbot’s coding potential.
TL;DR: GitHub Copilot handed two and failed two.
Check 1: Writing a WordPress Plugin
So, this failed miserably. This was my first check, so I can not inform but whether or not GitHub Copilot is horrible at writing code or whether or not the context by which one interacts with it’s limiting to the purpose the place it will probably’t meet this requirement.
Let me clarify.
This check includes asking the AI to create a totally purposeful WordPress plugin, full with admin interface parts and operational logic. The plugin takes in a set of names, types them, and, if there are duplicates, separates the duplicates so they are not facet by facet.
This was a real-world software that my spouse wanted as a part of an involvement machine she runs on her very energetic Fb group as a part of her digital items e-commerce enterprise.
A lot of the different AIs handed this check, at the least partly. 5 of the ten AI fashions examined handed the check fully. Three of them handed a part of the check. Two (together with Microsoft Copilot) failed fully.
The factor is, I gave GitHub Copilot the identical immediate I give all of them, nevertheless it solely wrote PHP code. To be clear, this downside might be solved solely utilizing PHP code. However some AIs like to incorporate some JavaScript for the interactive options. GitHub Copilot included code for utilizing JavaScript however by no means really generated the JavaScript that it tried to make use of.
What’s worse, once I created a JavaScript file and, from throughout the JavaScript file, tried to get GitHub Copilot to run the immediate, it gave me one other PHP script, which additionally referenced a JavaScript file.
As you’ll be able to see under, throughout the randomizer.js file, it tried to enqueue (principally to herald to run) the randomizer.js file, and the code it wrote was PHP, not JavaScript.
Check 2: Rewriting a string perform
This check is pretty easy. I wrote a perform that was supposed to check for {dollars} and cents however wound up solely testing for integers ({dollars}). The check asks the AI to repair the code.
GitHub Copilot did rework the code, however there have been a bunch of issues with the code it produced.
- It assumed a string worth was at all times a string worth. If it was empty, the code would break.
- The revised common expression code would break if a decimal level (i.e., “3.”) was entered, if a number one decimal level (i.e., “.3”) was entered, or if main zeros have been included (i.e., “00.30”).
For one thing that was supposed to check whether or not foreign money was entered appropriately, failing with code that may crash on edge instances just isn’t acceptable.
So, we have now one other fail.
Check 3: Discovering an annoying bug
GitHub Copilot acquired this proper. That is one other check pulled from my real-life coding escapades. What made this bug so annoying (and troublesome to determine) is that the error message is not instantly associated to the precise downside.
The bug is type of the coder equal of a trick query. Fixing it requires understanding how particular API calls within the WordPress framework work after which making use of that data to the bug in query.
Microsoft Copilot, Gemini, and Meta Code Llama all failed this check. However GitHub Copilot solved it appropriately.
Check 4: Writing a script
Right here, too, GitHub Copilot succeeded the place Microsoft Copilot failed. The problem right here is that I am testing the AI’s potential to create a script that is aware of about coding in AppleScript, the Chrome object mannequin, and a bit Mac-only third-party coding utility referred to as Keyboard Maestro.
To move this check, the AI has to have the ability to acknowledge that each one three coding environments want consideration after which tailor particular person traces of code to every of these environments.
Remaining ideas
On condition that GitHub Copilot makes use of GPT-4, I discover the truth that it failed half of the assessments discouraging. GitHub is nearly the most well-liked supply administration surroundings on the planet, and one would hope that the AI coding help was moderately dependable.
As with all issues AI, I am certain efficiency will get higher. Let’s keep tuned and test again in a couple of months to see if the AI is more practical at the moment.
Do you utilize an AI to assist with coding? What AI do you favor? Have you ever tried GitHub Copilot? Tell us within the feedback under.
You possibly can observe my day-to-day challenge updates on social media. Be sure you subscribe to my weekly replace publication, and observe me on Twitter/X at @DavidGewirtz, on Fb at Fb.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, on Bluesky at @DavidGewirtz.com, and on YouTube at YouTube.com/DavidGewirtzTV.
