<> How to use shell Script judgment hive Whether the database is imported dt Partitioned data ?
target ： take goals Judge all table names in , Is it hive of ods Middle layer
tmp="part-" # Only here "part-" This field # Traverse text file , Get table name for i in `cat /home/goals.txt` do
# obtain hdfs On the dt Content of time partition temp1=`hadoop fs -ls /idp/src/ods/${i}/dt=$1`
# Judge whether it is empty ： It's used here grep To find out temp1 Is there any in the string tmp("part-") This substring result=$(echo $temp1 | grep
"${temp0}") # Create a log of daily tasks cd /date touch $1.txt # Judge whether there is current day data , Enter the results into the created log if [[
"$result" != "" ]] then echo " yes " >> $1.txt else echo " no " >> $1.txt fi done
<> How to use python Remove the first line of all files in the file directory ?
1. Why python remove ?shell Scripts can also be implemented , See specific needs ,shell Suitable for single file ,python Easy to traverse the entire folder .
2. How to use the delete first line function ? Building block type , Put the traversal function found in the search together with the function of deleting the first line .
#conding=utf8 import fileinput import os # Get all files in the directory g = os.walk(r"/test") for
path,dir_list,file_list in g: for file_name in file_list: join = os.path.join(
path, file_name) # Delete the first line of the file one by one for line in fileinput.input(join, inplace=1): if
not fileinput.isfirstline(): print(line.replace('\n',''))
summary ： We are facing the challenge of writing programs at work ,1. You can try to improve the function first , For example, learn from the implementation methods of others on the network .2. Implement a single test , Do batch operation after completion .3. Finally, think about whether we can continue to further optimize . in short , We completed the development challenge by taking small steps and running fast .
Technology