Writing unit tests was already a headache for developers, and AI is making it worse. A recent study has unveiled a critical weakness in LLMs: their inability to create accurate unit tests. 

While ChatGPT and Copilot demonstrated impressive capabilities in generating correct code for simple algorithms (success rates ranging from 63% to 89%), their performance dropped significantly when tasked with producing unit tests which are used to evaluate production code.

ChatGPT’s test correctness fell to a mere 38% for Java and 29% for Python, with Copilot showing only slightly better results at 50% and 39%, respectively.

According to a study published by GitLab in 2023, automated test generation is one of the top use cases for AI in software development, with 41% of respondents currently using it. However, this recent study is now questioning the quality of those tests. 

A fullstack developer named Randy on Daily.dev forum mentioned that he had tried AI for both writing code and writing unit tests, and it failed miserably as it does not understand testing frameworks like Groovy and Spock.

If you practice TDD, LLMs shouldn’t be writing your tests (as good as it seems). You think through the use cases, break down the logic, and write the tests. Let the LLM fill in the implementation.

I wonder how far someone could get with this approach

— Andrew Nguonly (@andrewnguonly) June 2, 2024

Reason Why AI is Poor at Software Testing

AI-generated tests often lack the necessary context and understanding of specific requirements and nuances of a given codebase. Due to this, AI may result in an increase of “tautological testing” – tests that prove the code does what the code does rather than proving it’s doing what it’s supposed to do.

“It’s already a danger when you write the implementation first; AI is only going to make it worse,” a user explained in the Reddit discussion.

Moreover, relying on AI for test writing can lead to a false sense of security, as generated tests may not cover all critical scenarios, potentially compromising the software quality and reliability.

When an AI is asked to write unit tests for code that contains a bug, it typically doesn’t have the ability to identify that bug. Instead, it treats the existing code as the “correct” implementation and writes tests that validate the current behavior – including the bugs, if any.

Instead, the developer says that a better use for AI would be to ask it, “What are all the ways that this code can fail?” Instead of having it write tests, have it identify things you might have missed.

Another report by researchers from the University of Houston, suggested similar numbers as ChatGPT-3.5. Only 22.3% of generated tests were fully correct, and 62.3% were somewhat correct. 

Besides, the report noted that LLMs struggle to understand and write OpenMP and MPI unit tests due to the inherent complexity and domain-specific nature of parallel programming. Also, when provided with “too much” context, LLMs tended to hallucinate, generating code with nonexistent types, methods, and other constructs.

“Like other LLM-based tools, the generated tests are a “best guess” and developers shouldn’t blindly trust them. In many cases, additional debugging and editing are required,” said Ruiguo Yang, the founder of TestScribe. 

When developers consider making new test cases, AI still has a hard time doing that. With their creative problem-solving skills, human testers still need to make thorough test plans and define the overall testing scope.

But What is the Solution?

To solve this problem, researchers from the University of Houston used the LangChain memory method. They passed along smaller pieces of the code as a guide, allowing the system to fill in the rest, similar to how autocomplete works when you’re typing.

This proves that one of the most effective ways to tackle this problem is providing more context to the AI models, such as the full code or associated libraries, which significantly improves the compilation success rate. For instance, with ChatGPT, the increase was from 23.1% to 61.3%, and for Davinci, it was almost 80%.

In recent times, tools like Cursor are helping developers build code without any hassle and in future, we might see these tools building better unit tests along with production code. 

But for now, while AI can generate tests quickly, having an experienced engineer will remain crucial to assess the quality and usability of AI-generated code or tests.

The post Why AI Can’t Get Software Testing Right appeared first on AIM.

Notwithstanding that these technologies are still new and developing, there is almost no field where artificial intelligence or machine learning has not permeated in some form or another.

Learning about these rising technologies, therefore, becomes an important step to pushing your career forward and making groundbreaking innovations. We have compiled a list of a few courses launched in 2022 that freshers and professionals can pursue to march ahead in AI-ML.

1. MachineHack Pocket Courses

Unlike full-fledged machine learning courses that are time-consuming and resource intensive, MachineHack has a bunch of pocket courses covering various topics on machine learning and analytics helping individuals learn faster. and participate in hackathons, perfecting those skills with in-depth knowledge. 

Click here to check out individual courses on topics like linear regression, sequence modelling, image segmentation.

2. Machine learning specialisation

DeepLearning.AI and Stanford University Online have collaborated to create a beginner-friendly programme for teaching the fundamentals of machine learning and building real-world AI applications. It is a foundational specialisation with three courses taught by Andrew Ng, Aarti Bagul, Eddy Shyu, and Geoff Ladwig. The three-course programme is an updated version of Andrew’s machine learning course launched in 2012 and had over 4.8 million learners. 

The three courses included in the specialisation are — 

1. Supervised Machine Learning: Regression and Classification
2. Advanced Learning Algorithms
3. Unsupervised Learning, Recommenders, Reinforcement Learning

The course offers a broad introduction to modern machine learning using Python and other libraries like NumPy and scikit-learn. It encompasses topics such as building and training neural networks with TensorFlow, clustering and anomaly detection, building recommender systems, and building reinforcement learning models.

To check out the courses on Coursera, click here.

3. IIT Mandi Data Science Course

The Indian Institute of Technology, Mandi, has collaborated with National Skills Development Corporation to launch certificate courses on machine learning and data science starting in November. The courses will be conducted by the faculty members of IIT Mandi online. The programme will include foundational knowledge in data science and also specialise in machine learning with Python to allow learners to explore careers as data scientists, and business and data analysts. The course will last for nine months with certificates issued by IIT Mandi and NSDC.

To find out more about the course, click here.

4. Google AI Research YouTube 

Google AI announced their Google Research YouTube channel in August with focus on subjects like robotics, AI/ML, quantum computing, health and bioscience. The channel is a free resource for information about the subjects and includes three segments:

1. Meet a Google researcher, where they demonstrate their new innovations and technologies
2. ResearchBytes focuses on converting Google research publications into byte-sized content for easy understanding
3. Spotlights discusses new technologies by Google in-depth, focusing on individual innovations at a time
   

5. IIT Jodhpur’s PGDM in Data Science and Cloud Computing

In collaboration with WileyNXT, IIT Jodhpur launched a post-graduate diploma in cloud computing and data engineering. It is a 12-month intensive course teaching about managing the lifecycle of a data engineering project like big data collection and insight generation using machine learning techniques. The course requires a bachelor’s degree in science or engineering with a minimum of 50% marks. Along with this, the candidate must also be a working professional with a minimum of two years experience in the data science field.

Click here to learn more and apply for the course.

6. IIT Madras-Sony India Finishing School Programme

Pravartak Technologies of IIT Madras collaborated with Sony India Software Centre to launch a free course for pandemic-era kids with a family income less than INR 8 lakh. The course focuses on AI, ML, computer graphics, cybersecurity, and communication skills. Sony will offer employment to 15 top performers of the six-month course and the rest would be eligible to appear for job interviews at the IIT-M placement cell. The free course also offers a stipend to the top scoring candidates of the entrance examination. 

Click here to know more about the course.

7. Samsung Innovation Campus

Samsung India launched their Samsung Innovation Campus to upskill students and young professionals in AI, IoT, Big Data, and coding and prepare them for future opportunities. The first batch will be of 3,000 underprivileged students across India as proposed by Electronics Sector Skills Council of India (ESSCI). During the course, the students will receive mentorship from Samsung researchers and classes from ESSCI-approved educational experts. The AI course will be conducted for 270 hours of theory and 80 hours of project work.

To know more about the course, click here.

8. Fast.ai Practical Deep Learning for Coders

Fast.ai has been developing several courses for the past few years, but recently announced their Practical Deep Learning for Coders 2022 course for teaching learners about building deep learning models. The course includes computer vision, NLP, tabular analysis, and collaborative filtering problems. By the end of this course, students will learn how to implement fundamentals of deep learning like gradient descent, stochastic gradient, and complete training loop. The course is taught by Jeremy Howard with nine lessons, each about 90 minutes long. Alumni of previous editions of the course have been working at GoogleBrain, OpenAI, Amazon, Tesla, and Adobe.

Click here to know more about the programme.

The post Top AI Courses Launched in 2022 appeared first on AIM.

Analytics India Magazine spoke to Sayee Bellamkonda, a doctoral candidate at INSOFE-Rennes School of Business, who shares his experience of pursuing the leadership program in data science. 

Mr Bellamkonda is a Chief Digital and Technology Officer at Global Mobility Solutions (Vialto Partners, now part of PWC), a global company operating in 60 countries.

AIM: What motivated you to choose the Global DBA program from Rennes SB- INSOFE? 

Bellamkonda: I did my master’s in engineering followed by an MBA in Global Management. In leading teams at ITA organizations like American Express, Ameriprise, CBRE and Vialto Partners, the need for a practitioner’s view of Artificial Intelligence and Data Science became quite apparent. 

The DBA with Rennes School of Business (RSB) offered in collaboration with INSOFE is one of the only programs that met my requirement of combining data science with leadership skills and business management.

AIM: As a working professional, how was your experience taking this program?

Bellamkonda: The course is well planned to accommodate our schedules, and we don’t have any weekday classes. The way the content is shared is very convenient as I have access to class recordings, documents and relevant materials at all times. That being said, the program requires time commitment and effort.  

The DBA brings the best of both worlds – on one side, we have RSB with the best academic faculty, and on the other, we have the practitioners at INSOFE with tremendous experience. Also, the annual visit to Rennes in France is a good initiative that gets us face time with the faculty there.

I appreciate the time commitment from the INSOFE mentors, especially Dr Murthy and Dr Gnana (who joins in from Australia to teach our classes). 

AIM: How did this program help you with your career?

Bellamkonda: From my work at American Express, I have seen the potential of how meaningful insights can be obtained from data. Building an enterprise data platform, identifying the right data, and producing descriptive and predictive analytics have become crucial. Irrespective of the industry, it is important to have formal data science and artificial education to engage with such work. Going through this program gave me a new insight into how to address these problems for my wealth management clients in the global commercial real estate space.

AIM: Can you share some insights about your research journey?

Research Topic: Helping companies identify the right use cases which can be implemented using AI and ML frameworks.

Bellamkonda: My research topic is extremely relevant to all practitioners. There is a lot of investment in AI and ML spheres. By 2030, 70% of the companies will be using AI & ML to enhance business. But statistics predict that only 20% of these companies will be able to achieve the set goals. 

So how do we address this 80% shortfall? This is where my research comes into play. I work on building feasibility frameworks for companies. With this model, organizations will be able to look at various frameworks, recognize the knowledge gaps, and identify the right use cases to be implemented using AI & ML. This will drastically increase the success rate of organizations being able to implement AI & ML.

AIM: What has the DBA cohort experience been like?

Bellamkonda: My peers from Dubai and India greatly enhanced my learning experience. It was very interesting interacting with peers from diverse backgrounds, such as fashion technology, education, and supply chain management, as I come from a commercial real estate and financial management background. We share our experiences on a weekly basis and discuss the coursework, which helped us stay up to date. Here the learning happens not just from the professors but from peers as well.

It’s a culture that has been created in the three-year journey, which is more like a marathon than a solo sprint, where the cohorts motivate each other.

AIM: As you have completed a major part of the program, what key takeaways would you like to share with future aspirants?

Bellamkonda: Future aspirants interested in pursuing a doctoral program should know that this is an excellent opportunity to network and learn. Having access to strong support and experienced faculty is crucial to succeeding in a doctoral program. This DBA has all that and more.

Both RSB and INSOFE are subject area experts with highly knowledgeable professors. In addition, this program also brings together a global cohort of peers from diverse domains.

However, this is a doctoral program. It requires a time commitment and cannot be breezed through just by listening to lectures.

AIM: What are your thoughts on your final research project and patents, if any?

Bellamkonda: My first goal is to complete my doctoral program. Students of the DBA have been taught how to write patent descriptions, patent filing, and the dos and don’ts, which has been an extremely valuable experience. 

When my proposed research framework is built, it can be offered via open source sites for the world to learn. But it can also be productized. Rennes and INSOFE are given the opportunity to jointly patent these ideas, so we have equal commitment and responsibility in making the product successful. 

INSOFE is organising a free webinar on April 7, 2022, at 7.00 pm IST, to give an overview of the program and make a case for how a Doctoral degree can prepare you for strategic leadership roles in data science.

REGISTER HERE

The post The global doctorate programme for AI thought leadership appeared first on AIM.