The best way to Carry out Information Evaluation in Python Utilizing the OpenAI API — SitePoint

On this tutorial, you’ll learn to use Python and the OpenAI API to carry out knowledge mining and evaluation in your knowledge.

Manually analyzing datasets to extract helpful knowledge, and even utilizing easy applications to do the identical, can typically get sophisticated and time consuming. Fortunately, with the OpenAI API and Python it’s doable to systematically analyze your datasets for fascinating info with out over-engineering your code and losing time. This can be utilized as a common answer for knowledge evaluation, eliminating the necessity to use totally different strategies, libraries and APIs to research several types of knowledge and knowledge factors inside a dataset.

Let’s stroll by the steps of utilizing the OpenAI API and Python to research your knowledge, beginning with set issues up.

Setup

To mine and analyze knowledge by Python utilizing the OpenAI API, set up the openai and pandas libraries:

pip3 set up openai pandas

After you’ve performed that, create a brand new folder and create an empty Python file inside your new folder.

Analyzing Textual content Information

For this tutorial, I believed it could be fascinating to make Python analyze Nvidia’s newest earnings name.

Obtain the newest Nvidia earnings name transcript that I obtained from The Motley Idiot and transfer it into your undertaking folder.

Then open your empty Python file and add this code.

The code reads the Nvidia earnings transcript that you simply’ve downloaded and passes it to the extract_info operate because the transcript variable.

The extract_info operate passes the immediate and transcript because the person enter, in addition to temperature=0.3 and mannequin="gpt-3.5-turbo-16k". The rationale it makes use of the “gpt-3.5-turbo-16k” mannequin is as a result of it might probably course of massive texts akin to this transcript. The code will get the response utilizing the openai.ChatCompletion.create endpoint and passes the immediate and transcript variables as person enter:

completions = openai.ChatCompletion.create(
    mannequin="gpt-3.5-turbo-16k",
    messages=[
        {"role": "user", "content": prompt+"nn"+text}
    ],
    temperature=0.3,
)

The total enter will seem like this:

Extract the next info from the textual content: 
    Nvidia's income
    What Nvidia did this quarter
    Remarks about AI

Nvidia earnings transcript goes right here

Now, if we move the enter to the openai.ChatCompletion.create endpoint, the complete output will seem like this:

{
  "decisions": [
    {
      "finish_reason": "stop",
      "index": 0,
      "message": {
        "content": "Actual response",
        "role": "assistant"
      }
    }
  ],
  "created": 1693336390,
  "id": "request-id",
  "mannequin": "gpt-3.5-turbo-16k-0613",
  "object": "chat.completion",
  "utilization": {
    "completion_tokens": 579,
    "prompt_tokens": 3615,
    "total_tokens": 4194
  }
}

As you possibly can see, it returns the textual content response in addition to the token utilization of the request, which could be helpful for those who’re monitoring your bills and optimizing your prices. However since we’re solely within the response textual content, we get it by specifying the completions.decisions[0].message.content material response path.

For those who run your code, it’s best to get the same output to what’s quoted beneath:

From the textual content, we are able to extract the next info:

1. Nvidia’s income: Within the second quarter of fiscal 2024, Nvidia reported document Q2 income of 13.51 billion, which was up 88% sequentially and up 101% yr on yr.
2. What Nvidia did this quarter: Nvidia skilled distinctive development in varied areas. They noticed document income of their knowledge middle section, which was up 141% sequentially and up 171% yr on yr. Additionally they noticed development of their gaming section, with income up 11% sequentially and 22% yr on yr. Moreover, their skilled visualization section noticed income development of 28% sequentially. Additionally they introduced partnerships and collaborations with corporations like Snowflake, ServiceNow, Accenture, Hugging Face, VMware, and SoftBank.
3. Remarks about AI: Nvidia highlighted the robust demand for his or her AI platforms and accelerated computing options. They talked about the deployment of their HGX techniques by main cloud service suppliers and client web corporations. Additionally they mentioned the functions of generative AI in varied industries, akin to advertising, media, and leisure. Nvidia emphasised the potential of generative AI to create new market alternatives and enhance productiveness in numerous sectors.

As you possibly can see, the code extracts the information that’s specified within the immediate (Nvidia’s income, what Nvidia did this quarter, and remarks about AI) and prints it.

Analyzing CSV Information

Analyzing earnings-call transcripts and textual content information is cool, however to systematically analyze massive volumes of knowledge, you’ll must work with CSV information.

As a working instance, obtain this Medium articles CSV dataset and paste it into your undertaking file.

For those who have a look into the CSV file, you’ll see that it has the “creator”, “claps”, “reading_time”, “hyperlink”, “title” and “textual content” columns. For analyzing the medium articles with OpenAI, you solely want the “title” and “textual content” columns.

Create a brand new Python file in your undertaking folder and paste this code.

This code is a bit totally different from the code we used to research a textual content file. It reads CSV rows one after the other, extracts the required items of data, and provides them into new columns.

For this tutorial, I’ve picked a CSV dataset of Medium articles, which I obtained from HSANKESARA on Kaggle. This CSV evaluation code will discover the general tone and the principle lesson/level of every article, utilizing the “title” and “article” columns of the CSV file. Since I all the time come throughout clickbaity articles on Medium, I additionally thought it could be fascinating to inform it to seek out how “clickbaity” every article is by giving each a “clickbait rating” from 0 to three, the place 0 is not any clickbait and three is excessive clickbait.

Earlier than I clarify the code, analyzing your complete CSV file would take too lengthy and price too many API credit, so for this tutorial, I’ve made the code analyze solely the primary 5 articles utilizing df = df[:5].

You might be confused concerning the following a part of the code, so let me clarify:

for di in vary(len(df)):
    title = titles[di]
    summary = articles[di]
    additional_params = extract_info('Title: '+str(title) + 'nn' + 'Textual content: ' + str(summary))
    strive:
        end result = additional_params.break up("nn")
    besides:
        end result = {} 

This code iterates by all of the articles (rows) within the CSV file and, with every iteration, will get the title and physique of every article and passes it to the extract_info operate, which we noticed earlier. It then turns the response of the extract_info operate into a listing to separate the totally different items of information utilizing this code:

strive:
    end result = additional_params.break up("nn")
besides:
    end result = {} 

Subsequent, it provides each bit of information into a listing, and if there’s an error (if there’s no worth), it provides “No end result” into the listing:

strive:
    apa1.append(end result[0])
besides Exception as e:
    apa1.append('No end result')
strive:
    apa2.append(end result[1])
besides Exception as e:
    apa2.append('No end result')
strive:
    apa3.append(end result[2])
besides Exception as e:
    apa3.append('No end result')

Lastly, after the for loop is completed, the lists that comprise the extracted data are inserted into new columns within the CSV file:

df = df.assign(Tone=apa1)
df = df.assign(Main_lesson_or_point=apa2)
df = df.assign(Clickbait_score=apa3)

As you possibly can see, it provides the lists into new CSV columns which might be identify “Tone”, “Main_lesson_or_point” and “Clickbait_score”.

It then appends them to the CSV file with index=False:

df.to_csv("knowledge.csv", index=False)

The rationale why it’s a must to specify index=False is to keep away from creating new index columns each time you append new columns to the CSV file.

Now, for those who run your Python file, await it to complete and test our CSV file in a CSV file viewer, you’ll see the brand new columns, as pictured beneath.

For those who run your code a number of instances, you’ll discover that the generated solutions differ barely. It is because the code makes use of temperature=0.3 so as to add a little bit of creativity into its solutions, which is helpful for subjective matters like clickbait.

Working with A number of Information

If you wish to mechanically analyze a number of information, you want to first put them inside a folder and ensure the folder solely accommodates the information you’re considering, to forestall your Python code from studying irrelevant information. Then, set up the glob library utilizing pip3 set up glob and import it in your Python file utilizing import glob.

In your Python file, use this code to get a listing of all of the information in your knowledge folder:

data_files = glob.glob("data_folder/*")

Then put the code that does the evaluation in a for loop:

for i in vary(len(data_files)):

Contained in the for loop, learn the contents of every file like this for textual content information:

f = open(f"data_folder/{data_files[i]}", "r")
txt_data = f.learn()

Additionally like this for CSV information:

df = pd.read_csv(f"data_folder/{data_files[i]}")

As well as, be sure to avoid wasting the output of every file evaluation right into a separate file utilizing one thing like this:

df.to_csv(f"output_folder/knowledge{i}.csv", index=False)

Conclusion

Keep in mind to experiment together with your temperature parameter and regulate it to your use case. If you need the AI to make extra artistic solutions, enhance your temperature, and if you need it to make extra factual solutions, be sure to decrease it.

The mixture of OpenAI and Python knowledge evaluation has many functions aside from article and earnings name transcript evaluation. Examples embrace information evaluation, e-book evaluation, buyer overview evaluation, and rather more! That mentioned, when testing your Python code on huge datasets, be sure to solely take a look at it on a small a part of the complete dataset to avoid wasting API credit and time.