Parsing Text Files with Regular Expression

Title       : CRB: Genetic Diversity of Endangered Populations of Mysticete Whales:
               Mitochondrial DNA and Historical Demography
Type        : Award
NSF Org     : DEB 
Latest
Amendment
Date        : August 1,  1991     
File        : a9000006

Award Number: 9000006
Award Instr.: Continuing grant                             
Prgm Manager: Scott Collins                           
	      DEB  DIVISION OF ENVIRONMENTAL BIOLOGY       
	      BIO  DIRECT FOR BIOLOGICAL SCIENCES          
Start Date  : June 1,  1990       
Expires     : November 30,  1992   (Estimated)
Expected
Total Amt.  : $179720             (Estimated)
Investigator: Stephen R. Palumbi   (Principal Investigator current)
Sponsor     : U of Hawaii Manoa
	      2530 Dole Street
	      Honolulu, HI  968222225    808/956-7800

NSF Program : 1127      SYSTEMATIC & POPULATION BIOLO
Fld Applictn: 0000099   Other Applications NEC                  
              61        Life Science Biological                 
Program Ref : 9285,
Abstract    :
                                                                                             
              Commercial exploitation over the past two hundred years drove                  
              the great Mysticete whales to near extinction.  Variation in                   
              the sizes of populations prior to exploitation, minimal                        
              population size during exploitation and current population                     
              sizes permit analyses of the effects of differing levels of                    
              exploitation on species with different biogeographical                         
              distributions and life-history characteristics.  Dr. Stephen                   
              Palumbi at the University of Hawaii will study the genetic                     
              population structure of three whale species in this context,                   
              the Humpback Whale, the Gray Whale and the Bowhead Whale.  The                 
              effect of demographic history will be determined by comparing                  
              the genetic structure of the three species.  Additional studies                
              will be carried out on the Humpback Whale.  The humpback has a                 
              world-wide distribution, but the Atlantic and Pacific                          
              populations of the northern hemisphere appear to be discrete                   
              populations, as is the population of the southern hemispheric                  
              oceans.  Each of these oceanic populations may be further                      
              subdivided into smaller isolates, each with its own migratory                  
              pattern and somewhat distinct gene pool.  This study will                      
              provide information on the level of genetic isolation among                    
              populations and the levels of gene flow and genealogical                       
              relationships among populations.  This detailed genetic                        
              information will facilitate international policy decisions                     
              regarding the conservation and management of these magnificent                 
              mammals.

import pandas as pd
import nltk
import re
import os

# get a list of all the files in the directory
all_files = os.listdir("data/")

# create a new pandas dataframe to fill
new_df = pd.DataFrame(columns=['file','org','amt','abs'])

# iterate over the list of files, opening each one and extracting info
#  - then appending it to the dataframe created above
for txt_file in all_files:
    with open('data/'+txt_file, encoding='utf-8', errors='ignore') as f:
        read_data = f.read()
    # regex text selections
    nsv_file = re.findall('(?<=File        : ).*',read_data)
    nsv_org = re.findall('(?<=NSF Org     : ).*',read_data)
    nsv_amt = re.findall('(?<=Total Amt.  : ).*',read_data)
    nsv_abs = re.findall('(?<=Abstract    :)[\s\S]*',read_data)
    # append data to dataframe
    new_df = new_df.append({'file' : nsv_file, 'org' : nsv_org, 'amt' : nsv_amt, 'abs':nsv_abs},ignore_index=True)

# save as csv
new_df.to_csv('tabular_dataset.csv')

# View the dataframe shape and some data, make sure all data was loaded
print("Rows, Columns : ",new_df.shape)
new_df.head()

Rows, Columns :  (4017, 4)

# read in csv as pandas dataframe, prevents type issues
data_set = pd.read_csv('tabular_dataset.csv')

# Clean dataset of newlines (\n), (Estimated), and extra whitespace using regex replace
data_set = data_set.replace(r'\\n',' ', regex=True) 
data_set = data_set.replace(r'\(Estimated\)',' ', regex=True)
data_set = data_set.replace(r' +',' ', regex=True) 
# Check out the first few rows of data
data_set.head()

# split the abstract (abs) column and append it as new column called split
data_set['split'] = data_set['abs'].str.split(".")

# take another quick peek at the data after appending the column and splitting abs
data_set.head()

# set row_choice to whatever row you want to analyze the abstract of
row_ch = 1

# iterate through the row choice abstract, print relevant information
iterator = 0
for line in data_set['split'][row_ch]:
    print(data_set['file'][row_ch]," | ", iterator, " | ", line,"\n")
    iterator += 1
# Print the final number of sentences
print("Number of sentences: ", len(data_set['split'][row_ch]))

['a9009851']  |  0  |  [' Pyruvate and phosphoenol-pyruvate (PEP) are two central intermediates in cellular metabolism 

['a9009851']  |  1  |   They are the branch points of many catabolic and biosynthetic pathways 

['a9009851']  |  2  |   Although the importance of these intermediates has long been recognized, the use of both genetic and engineering techniques for studying the physiological effect of redirected pyruvate and PEP metabolism has not been reported 

['a9009851']  |  3  |   Pyruvate is not normally recycled back to PEP under glycolytic conditions 

['a9009851']  |  4  |   Because of this irreversibility, the yields of many specialty chemicals produced from glucose via bacterial fermentations remain low 

['a9009851']  |  5  |   The Principal Investigator proposes to construct and characterize strains of the bacteria E 

['a9009851']  |  6  |   coli which can recycle pyruvate back to PEP 

['a9009851']  |  7  |   In addition to improving the yields of amino acid fermentations, the results of this project could contribute to the understanding of how the cell distributes its carbon source, and how to decouple product formation from cell growth 

['a9009851']  |  8  |   '] 

Number of sentences:  9

Related Posts

• Visual Analysis of Fishing Data with Python
• Forecasting Market Reaction Using Twitter
• Insights from Wardriving Data
• Extending Data with APIs