# PY3. Pandas - Data Manipulation and Analysis

### Introduction

Python's Pandas package is used to manipulate data collections. It offers tools for data exploration, cleaning, analysis, and manipulation. Wes McKinney came up with the name "Pandas" in 2008, and it refers to both "Panel Data" and "Python Data Analysis.”

### How to Install Pandas

To install Pandas, first, make sure that Python and Pip is already installed in your system. If they are installed, then you can install Pandas just by running this command on the command line.

```python
>> pip install pandas
```

You can use other Python distributions like Spyder or Anaconda where pandas are already installed.

### Write your first code using Pandas

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620049667/f2532b11-2d57-46d4-a37d-ded8eed17708.png align="left")

To use Pandas in our code, first, we have to import it. Here we are importing pandas as `pd`. Here, `pd` is an alias for `panda`.

### Series

A series in pandas is like a column in a table that can hold data like a one-dimensional array of any type. To create a Pandas series, we can write this code:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620052989/9ad3e966-26a9-4e94-8af3-594eee9fcb5f.png align="left")

Here we have used a method called `Series()` to convert a list into a series. After that, we can print the series using `print()` function. If we observe the output, we will see that there are 5 rows and 2 columns. In the end, it shows the data type of the elements in the list that we have converted into a series. The first column is the column of labels. If nothing else is specified, the values are labeled with their index number. The first value has an index of 0, the second value has an index of 1, etc. This label can be used to access a specified value.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620055070/87b3ca12-4526-4593-b64d-8f65fbb1dc14.png align="left")

Now, it's clear that we can use the labels as an index and access a specific value using it. The fun thing is, that we can name our own labels using the `index` argument.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620058104/70ab7451-65da-4422-b625-d80a8dcfa5d3.png align="left")

After creating our labels, we can access an item by referring to the label like we access a dictionary value using the key:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620059588/f6e7ad7a-074d-41fd-93a6-808a85d3b219.png align="left")

**Key/Value Objects as Series**

You can also use a key-value object, like a dictionary when creating a series.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620060824/6925179c-e69b-4236-86db-0e9a3f3f9f9a.png align="left")

Use the index option to specify only the words you wish to be included in the Series, leaving out the rest of the words in the dictionary.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620063061/5b0f4c04-375f-4c8a-9292-ce979ea6a3c5.png align="left")

Now let's see what happens when we add two series.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620065830/8a5be181-9cc2-46be-8259-1f06c76eb9d4.png align="left")

It is creating a new series by adding the same labeled data together and converting them into floats. If the same index is missing, then the output will be `NaN`.

### **DataFrames**

A Pandas DataFrame is a 2-dimensional data structure, like a 2-dimensional array, or a table with rows and columns. To understand it better, first, let’s compare it with the pandas series.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620067326/b00546d3-adbb-4879-b343-21bbc2a137fb.png align="left")

The first visible difference here is that a series has a specific data type, but a data frame doesn’t.

**Dictionary and Data Frame**

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620070353/7df93f3b-fb23-4865-8a43-19618052f266.png align="left")

Here we can see that the keys of the dictionary are used as the titles of the columns, and the lists are the data of the columns. Now, what is happening here is that we are passing a list and it is creating a data frame. It also defines its index and column names on its own. If we want to use our own name as an index or column name, we have to pass it as an argument through the `DataFrame()` method.

**Generate Data Frame**

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620073851/31ac4515-9044-4314-b8ec-b5ae7a0e0eb2.png align="left")

In the example above, we are passing a 2D array, index, and column name. After that, the `DataFrame()` method automatically allocates the index and column name.

**Selection and Indexing**

Firstly, DataFrame columns are just Series:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620075325/85fbc3e4-37ef-4112-8181-dd5400b8cdf3.png align="left")

Now, let’s learn the various methods to grab data from a DataFrame

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620076636/704d2c8e-6238-4a9d-8c57-7b13a6614fe7.png align="left")

We can grab any column by using the column name. To access multiple columns, we have to pass a list of column names like this:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620079082/a2f5bea8-9802-4c1f-89a1-59b56c4cb332.png align="left")

We can also use the SQL Syntax but it is not recommended as it creates confusion:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620083620/df884e11-5efe-45ea-8844-4dd7feb29ca0.png align="left")

**Creating a new column:**

We can add a new column to our DataFrame like below. We can assign either an Array or Series to define the new column. In the example below, we are adding two columns, which are basically two series, and assigning a new column named `‘new’`:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620086418/72f3ff53-2b26-4b25-8b7f-6fd9480dca1a.png align="left")

**Removing Columns**

We have to use`.drop()` a method to drop a column or row. Here `axis=0` means row and `axis=1` means column. We have to mention it if we want to remove the column. The default value is `0`.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620089471/351ac951-c349-4ead-ada3-aad971489f39.png align="left")

But there is a problem. If we try to print `df` again, we will see that column `new` is still there.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620091814/09b518c8-daff-4b3a-813b-28c23d2b57a8.png align="left")

But why? Because to remove the column completely from the DataFrame, we have to use the `inplace` attribute. The Python developers kept that feature so that we do not have to face unwanted data loss.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620094161/76bd6ab9-cd2b-4f07-9c8e-ea493d67b3e5.png align="left")

Congratulations! The `new` column is completely removed from the data frame. We can also drop a row by changing the `axis` from `1` to `0`:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620096551/cb6208b5-c6d5-4c4a-be37-8148e2bf75f6.png align="left")

**Locate Rows**

We already know that a data frame is like a table with rows and columns. If we want to display or store a specific row, we have to use the `loc` method:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620098787/eb10ac96-9435-45fa-91f1-d50bfd17667c.png align="left")

Alternatively, you can choose based on position rather than the label using `iloc`:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620100779/40dc9907-d320-4cde-87cb-aa8beda9d488.png align="left")

We can also select a subset of rows and columns by the following method:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620101960/fbac9998-f48f-49f4-afb8-da379711fe4d.png align="left")

**Conditional Selection**

An important feature of pandas is conditional selection using bracket notation, very similar to Numpy:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620103951/fa4636c9-ddb9-4e23-a427-cd16ff729226.png align="left")

Here we can see a truth table based on the conditions we have given. When a cell satisfies our conditions, it's giving `True`. Otherwise, it's giving `false`. Now, if we want to print the value instead of true or false, we can write our code like this:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620105733/2f7eeb36-241c-4081-8852-0b176479f8c8.png align="left")

Now, it is printing only the values that satisfy our conditions.

What if our condition is only based on a specific column and we want to print the values according to the data of that column? In that case, we have to mention that specific column using this notation:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620107800/97157828-3a75-4975-9f4c-9b158d031627.png align="left")

For printing a specific column of our new filtered data frame, we just have to mention this like this:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620109541/1c9151dc-deef-4506-b047-34a6e2ede59d.png align="left")

For two conditions, you can use `|` and `&` with parenthesis:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620110910/ddb4ea78-5457-4eb2-98d4-35b811b2c8de.png align="left")

**More Index Details** Let's discuss some more features of indexing, including resetting the index or setting it to something else. We'll also talk about index hierarchy!

By using the `.split()` function, we can create a new column. It will create a list, and then we will create a new column using that:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620113372/e084c878-ab92-4b35-8d60-005e0b853a5c.png align="left")

We can also set an existing column as an index by using the `set_index()` method and passing the name of the column through it:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620115619/c338593e-a989-450f-8fe3-0b3fe185d4e1.png align="left")

Again, we have to use the `inplace` argument if we want to make the change permanent.

**Missing Data**

Let's show a few convenient methods to deal with missing data in pandas:

First, let’s create a new data frame using a dictionary:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620118295/12421d44-2f2a-492e-8aef-be6c5d98dc48.png align="left")

Here what we are doing is, we are intentionally putting some `nan` data to our data frame and assuming that those are missing data.

If we want to remove those missing values from our data frame, there is a method called `.dropna()`. It will remove all the rows that contain at least one missing data point. We can use the `asix` argument if we check the missing data column-wise.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620120261/e6aa7a58-ef69-4441-8bfd-fbe4a4b38481.png align="left")

We can use the `thresh` attribute if we need to keep some missing values by mentioning how many missing values we will consider:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620124975/ad79282b-2a00-46b8-a3f7-19bf9db9dc64.png align="left")

Here, `thresh=2` means - keep only the rows with at least 2 non-NA values.

If we want to fill our missing values with something else, we have to write our code like that by using the `.fillna()` method:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620127245/145d5f89-9458-4ee1-868b-dd26d1d60a84.png align="left")

Sometimes we have to fill in our missing data using a mean value. For that, we can code like this:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620129034/4493d1f5-6618-4ee4-81fd-72ea3785e9d9.png align="left")

**Groupby**

Groupby allows you to group together rows based on a column and perform an aggregate function on them. Here aggregate function means a function that takes some data and returns it as the sum of those data or the mean value of those data.

Firstly, let us create a data frame using a dictionary:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620130402/83a4efaa-183e-4245-985b-c3ba392d6228.png align="left")

Now you can use the `.groupby()` method to group rows together based on a column name. For instance, let's group based on `company`. This will create a `DataFrameGroupBy` object:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620132349/f6316f1c-c6b4-48c9-b0b7-a5e96aff49a2.png align="left")

You can save this object as a new variable:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620133775/878788e4-97e7-4ed7-8a08-1f2f3547cfb7.png align="left")

And then call aggregate methods on the object:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620136404/648c60d1-b89d-403b-9e2c-953500fd5426.png align="left")

Here, we are using the variable to call aggregate methods on the object. We can also do this directly like this:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620138516/0e4eae4d-0e4c-4e91-9ce7-1cb758f7f049.png align="left")

**More examples of aggregate methods:**

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620143893/da3493d7-2690-4e2d-8bcd-6e0aa899dc84.png align="left")

You can also do things such as max and min.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620146850/64112cad-5902-4593-a10d-83c36b7f282e.png align="left")

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620149021/a0a13163-0f94-4bb5-aec1-1a9565152e0f.png align="left")

Some other useful aggregate functions that you may find yourself doing are things such as count which just counts the number of instances or columns. In this case, it was able to return the person column because it's able to count how many instances of a person occur in that column or company. So we have two people and they have two sales each and that makes sense.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620150855/75305144-0cac-47e1-b08e-50b4a93c7b36.png align="left")

One last useful thing I want to show you with `groupby` is the `describe()` method and that gives you a bunch of useful information all at once.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620152954/333e6619-249a-4455-b425-379af7f5580c.png align="left")

And if you don't like this format, you can actually transpose this. So, you can say something like:

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620154707/f0a918ce-56fd-452c-8f9f-7490d5e82fca.png align="left")

So, whatever format you like better you can describe to that and then you can actually just call column names of this if you're just interested in a single column.

![Python Pandas](https://cdn.hashnode.com/res/hashnode/image/upload/v1699620156602/1756c9d2-f3b0-4c70-8a42-1030cc91597a.png align="left")