Sujet Précédent Sujet Suivant

Pig Latin

Fermé
Chercheur000 Messages postés 5 Date d'inscription dimanche 18 décembre 2016 Statut Membre Dernière intervention 20 décembre 2016 - Modifié par Chercheur000 le 18/12/2016 à 22:30
mamiemando Messages postés 33108 Date d'inscription jeudi 12 mai 2005 Statut Modérateur Dernière intervention 14 mai 2024 - 21 déc. 2016 à 12:01
Bonjour,

Dans l'execution du programme "Compteur de mots" version Pig en mode local, toutes les commandes marchent bien sauf "store" donne l'erreur suivante:


HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.6.0 0.15.0 hduser 2016-12-18 16:04:45 2016-12-18 16:04:52 GROUP_BY

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_local1472906397_0003 A,B,C,D,E GROUP_BY,COMBINER Message: Job failed! file:///home/soukaina/Bureau/results,

Input(s):
Failed to read data from "file:///home/soukaina/Bureau/hdfile"

Output(s):
Failed to produce result in "file:///home/soukaina/Bureau/results"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local1472906397_0003


2016-12-18 16:04:52,668 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2016-12-18 16:04:52,760 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job job_local1472906397_0003 failed, hadoop does not return any error message
2016-12-18 16:04:52,775 [main] WARN org.apache.pig.tools.grunt.GruntParser - There is no log file to write to.
2016-12-18 16:04:52,776 [main] ERROR org.apache.pig.tools.grunt.GruntParser - org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job job_local1472906397_0003 failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:179)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:631)
at org.apache.pig.Main.main(Main.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

2016-12-18 16:04:55,059 [main] INFO org.apache.pig.Main - Pig script completed in 58 seconds and 929 milliseconds (58929 ms)
2016-12-18 16:04:57,285 [communication thread] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce

& Voici le programme que j'ai exécuté via la commende pig -x local WordCountLocal.pig

A = LOAD 'hdfile' USING TextLoader AS ligne:chararray;
DESCRIBE A;
B = FOREACH A GENERATE TOKENIZE(LOWER(ligne)) AS mots;
DESCRIBE B;
C = FOREACH B GENERATE FLATTEN(mots) AS mot;
DUMP C;
D = GROUP C BY mot;
DUMP D;
DESCRIBE D;
E = FOREACH D GENERATE group AS mot, COUNT(C) as occurrences;
DESCRIBE E;
STORE E INTO 'results';

merci.

2 réponses

Réponse 1 / 2
mamiemando Messages postés 33108 Date d'inscription jeudi 12 mai 2005 Statut Modérateur Dernière intervention 14 mai 2024 7 753
19 déc. 2016 à 10:27
Bonjour,

Si j'en crois ton message d'erreur : 

Failed to read data from "file:///home/soukaina/Bureau/hdfile"


... j'en déduis que parmi les explications possibles, il y a :
- le chemin vers le fichier d'entrée est faux
- le fichier n'a pas des droits suffisants pour être lu par le profil utilisateur qui lance l'application que tu mentionnes
- le fichier est corrompu ou n'a pas le format attendu par ton logiciel.

Bonne chance
0
Chercheur000 Messages postés 5 Date d'inscription dimanche 18 décembre 2016 Statut Membre Dernière intervention 20 décembre 2016
19 déc. 2016 à 21:43
mais les commandes DUMP C & DUMP D ont retourné le resultat attendu c'est à dire le probléme n'est qu'au niveau de la commande STORE E
0
Chercheur000 Messages postés 5 Date d'inscription dimanche 18 décembre 2016 Statut Membre Dernière intervention 20 décembre 2016
19 déc. 2016 à 21:57
2016-12-19 20:50:36,636 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-12-19 20:50:36,657 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.6.0 0.15.0 hduser 2016-12-19 20:50:24 2016-12-19 20:50:36 GROUP_BY

Success!

Job Stats (time in seconds):
JobId Maps Reduces MaxMapTime MinMapTime AvgMapTime MedianMapTime MaxReduceTime MinReduceTime AvgReduceTime MedianReducetime Alias Feature Outputs
job_local2023858608_0002 1 1 n/a n/a n/a n/a n/a n/a n/a n/a A,B,C,D GROUP_BY file:/tmp/temp766825122/tmp1271703645,

Input(s):
Successfully read 4 records from: "file:///home/soukaina/Bureau/hdfile"

Output(s):
Successfully stored 7 records in: "file:/tmp/temp766825122/tmp1271703645"

Counters:
Total records written : 7
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local2023858608_0002


2016-12-19 20:50:36,702 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-19 20:50:36,707 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-19 20:50:36,726 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-19 20:50:36,822 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Success!
2016-12-19 20:50:36,857 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-12-19 20:50:36,859 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-12-19 20:50:36,860 [main] WARN org.apache.pig.data.SchemaTupleBackend - SchemaTupleBackend has already been initialized
2016-12-19 20:50:37,123 [main] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2016-12-19 20:50:37,124 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
(je,{(je),(je),(je)})
(à,{(à)})
(fes,{(fes)})
(suis,{(suis),(suis),(suis)})
(contente,{(contente)})
(j'habite,{(j'habite)})
(soukaina,{(soukaina),(soukaina)})
D: {group: chararray,C: {(mot: chararray)}}
E: {mot: chararray,occurrences: long}
2016-12-19 20:50:38,122 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-12-19 20:50:38,134 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-12-19 20:50:38,174 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
2016-12-19 20:50:38,704 [main] INFO org.apache.pig.tools.pigstats.ScriptState - Pig features used in the script: GROUP_BY
2016-12-19 20:50:39,074 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-12-19 20:50:39,075 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-12-19 20:50:39,075 [main] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2016-12-19 20:50:39,076 [main] INFO org.apache.pig.newplan.logical.optimizer.LogicalPlanOptimizer - {RULES_ENABLED=[AddForEach, ColumnMapKeyPrune, ConstantCalculator, GroupByConstParallelSetter, LimitOptimizer, LoadTypeCastInserter, MergeFilter, MergeForEach, PartitionFilterOptimizer, PredicatePushdownOptimizer, PushDownForEachFlatten, PushUpFilter, SplitFilter, StreamTypeCastInserter]}
2016-12-19 20:50:39,099 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MRCompiler - File concatenation threshold: 100 optimistic? false
2016-12-19 20:50:39,124 [main] INFO org.apache.pig.backend.hadoop.executionengine.util.CombinerOptimizerUtil - Choosing to move algebraic foreach to combiner
2016-12-19 20:50:39,273 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size before optimization: 1
2016-12-19 20:50:39,274 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MultiQueryOptimizer - MR plan size after optimization: 1
2016-12-19 20:50:39,329 [main] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-12-19 20:50:39,331 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-19 20:50:39,336 [main] INFO org.apache.pig.tools.pigstats.mapreduce.MRScriptState - Pig script settings are added to the job
2016-12-19 20:50:39,339 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - mapred.job.reduce.markreset.buffer.percent is not set, set to default 0.3
2016-12-19 20:50:39,340 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Reduce phase detected, estimating # of required reducers.
2016-12-19 20:50:39,341 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
2016-12-19 20:50:39,374 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator - BytesPerReducer=1000000000 maxReducers=999 totalInputFileSize=67
2016-12-19 20:50:39,375 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting Parallelism to 1
2016-12-19 20:50:39,431 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler - Setting up single store job
2016-12-19 20:50:39,440 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Key [pig.schematuple] is false, will not generate code.
2016-12-19 20:50:39,441 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Starting process to move generated code to distributed cacche
2016-12-19 20:50:39,441 [main] INFO org.apache.pig.data.SchemaTupleFrontend - Distributed cache not supported or needed in local mode. Setting key [pig.schematuple.local.dir] with code temp directory: /tmp/1482180639439-0
2016-12-19 20:50:39,738 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 1 map-reduce job(s) waiting for submission.
2016-12-19 20:50:39,762 [JobControl] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-19 20:50:40,148 [JobControl] WARN org.apache.hadoop.mapreduce.JobSubmitter - No job jar file set. User classes may not be found. See Job or Job#setJar(String).
2016-12-19 20:50:40,225 [JobControl] INFO org.apache.hadoop.mapreduce.lib.input.FileInputFormat - Total input paths to process : 1
2016-12-19 20:50:40,231 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths to process : 1
2016-12-19 20:50:40,234 [JobControl] INFO org.apache.pig.backend.hadoop.executionengine.util.MapRedUtil - Total input paths (combined) to process : 1
2016-12-19 20:50:40,680 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - number of splits:1
2016-12-19 20:50:41,022 [JobControl] INFO org.apache.hadoop.mapreduce.JobSubmitter - Submitting tokens for job: job_local1354877940_0003
2016-12-19 20:50:41,896 [JobControl] INFO org.apache.hadoop.mapreduce.Job - The url to track the job: http://localhost:8080/
2016-12-19 20:50:41,901 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - HadoopJobId: job_local1354877940_0003
2016-12-19 20:50:41,902 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Processing aliases A,B,C,D,E
2016-12-19 20:50:41,902 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - detailed locations: M: A[1,4],B[-1,-1],C[5,4],E[10,4],D[7,4] C: E[10,4],D[7,4] R: E[10,4]
2016-12-19 20:50:41,944 [Thread-50] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter set in config null
2016-12-19 20:50:41,949 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 0% complete
2016-12-19 20:50:41,950 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local1354877940_0003]
2016-12-19 20:50:41,996 [Thread-50] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.textoutputformat.separator is deprecated. Instead, use mapreduce.output.textoutputformat.separator
2016-12-19 20:50:42,006 [Thread-50] INFO org.apache.hadoop.conf.Configuration.deprecation - fs.default.name is deprecated. Instead, use fs.defaultFS
2016-12-19 20:50:42,006 [Thread-50] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.reduce.markreset.buffer.percent is deprecated. Instead, use mapreduce.reduce.markreset.buffer.percent
2016-12-19 20:50:42,007 [Thread-50] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.reduce.tasks is deprecated. Instead, use mapreduce.job.reduces
2016-12-19 20:50:42,009 [Thread-50] INFO org.apache.hadoop.conf.Configuration.deprecation - mapred.job.tracker is deprecated. Instead, use mapreduce.jobtracker.address
2016-12-19 20:50:42,011 [Thread-50] INFO org.apache.hadoop.mapred.LocalJobRunner - OutputCommitter is org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputCommitter
2016-12-19 20:50:42,037 [Thread-50] ERROR org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter - Mkdirs failed to create file:/home/soukaina/Bureau/results/_temporary/0
2016-12-19 20:50:42,047 [Thread-50] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for map tasks
2016-12-19 20:50:42,051 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local1354877940_0003_m_000000_0
2016-12-19 20:50:42,094 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorProcessTree : [ ]
2016-12-19 20:50:42,116 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Processing split: Number of splits :1
Total Length = 67
Input split[0]:
Length = 67
ClassName: org.apache.hadoop.mapreduce.lib.input.FileSplit
Locations:

-----------------------

2016-12-19 20:50:42,154 [LocalJobRunner Map Task Executor #0] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigRecordReader - Current split being processed file:/home/soukaina/Bureau/hdfile:0+67
2016-12-19 20:50:42,786 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - (EQUATOR) 0 kvi 26214396(104857584)
2016-12-19 20:50:42,786 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - mapreduce.task.io.sort.mb: 100
2016-12-19 20:50:42,793 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - soft limit at 83886080
2016-12-19 20:50:42,793 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - bufstart = 0; bufvoid = 104857600
2016-12-19 20:50:42,794 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - kvstart = 26214396; length = 6553600
2016-12-19 20:50:42,818 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
2016-12-19 20:50:42,873 [LocalJobRunner Map Task Executor #0] INFO org.apache.pig.data.SchemaTupleBackend - Key [pig.schematuple] was not set... will not generate code.
2016-12-19 20:50:42,950 [LocalJobRunner Map Task Executor #0] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapReduce$Map - Aliases being processed per job phase (AliasName[line,offset]): M: A[1,4],B[-1,-1],C[5,4],E[10,4],D[7,4] C: E[10,4],D[7,4] R: E[10,4]
2016-12-19 20:50:42,983 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner -
2016-12-19 20:50:42,985 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Starting flush of map output
2016-12-19 20:50:42,989 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Spilling map output
2016-12-19 20:50:42,990 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - bufstart = 0; bufend = 151; bufvoid = 104857600
2016-12-19 20:50:42,990 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - kvstart = 26214396(104857584); kvend = 26214352(104857408); length = 45/6553600
2016-12-19 20:50:43,098 [LocalJobRunner Map Task Executor #0] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigCombiner$Combine - Aliases being processed per job phase (AliasName[line,offset]): M: A[1,4],B[-1,-1],C[5,4],E[10,4],D[7,4] C: E[10,4],D[7,4] R: E[10,4]
2016-12-19 20:50:43,118 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.MapTask - Finished spill 0
2016-12-19 20:50:43,135 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.Task - Task:attempt_local1354877940_0003_m_000000_0 is done. And is in the process of committing
2016-12-19 20:50:43,147 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner - map
2016-12-19 20:50:43,150 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.Task - Task 'attempt_local1354877940_0003_m_000000_0' done.
2016-12-19 20:50:43,152 [LocalJobRunner Map Task Executor #0] INFO org.apache.hadoop.mapred.LocalJobRunner - Finishing task: attempt_local1354877940_0003_m_000000_0
2016-12-19 20:50:43,156 [Thread-50] INFO org.apache.hadoop.mapred.LocalJobRunner - map task executor complete.
2016-12-19 20:50:43,159 [Thread-50] INFO org.apache.hadoop.mapred.LocalJobRunner - Waiting for reduce tasks
2016-12-19 20:50:43,164 [pool-8-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - Starting task: attempt_local1354877940_0003_r_000000_0
2016-12-19 20:50:43,287 [pool-8-thread-1] INFO org.apache.hadoop.mapred.Task - Using ResourceCalculatorProcessTree : [ ]
2016-12-19 20:50:43,298 [pool-8-thread-1] INFO org.apache.hadoop.mapred.ReduceTask - Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@15517d3
2016-12-19 20:50:43,333 [pool-8-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - MergerManager: memoryLimit=709551680, maxSingleShuffleLimit=177387920, mergeThreshold=468304128, ioSortFactor=10, memToMemMergeOutputsThreshold=10
2016-12-19 20:50:43,355 [EventFetcher for fetching Map Completion Events] INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher - attempt_local1354877940_0003_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
2016-12-19 20:50:43,390 [localfetcher#2] INFO org.apache.hadoop.mapreduce.task.reduce.LocalFetcher - localfetcher#2 about to shuffle output of map attempt_local1354877940_0003_m_000000_0 decomp: 110 len: 114 to MEMORY
2016-12-19 20:50:43,415 [localfetcher#2] INFO org.apache.hadoop.mapreduce.task.reduce.InMemoryMapOutput - Read 110 bytes from map-output for attempt_local1354877940_0003_m_000000_0
2016-12-19 20:50:43,422 [localfetcher#2] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - closeInMemoryFile -> map-output of size: 110, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->110
2016-12-19 20:50:43,476 [EventFetcher for fetching Map Completion Events] INFO org.apache.hadoop.mapreduce.task.reduce.EventFetcher - EventFetcher is interrupted.. Returning
2016-12-19 20:50:43,508 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 50% complete
2016-12-19 20:50:43,514 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Running jobs are [job_local1354877940_0003]
2016-12-19 20:50:43,516 [pool-8-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied.
2016-12-19 20:50:43,535 [pool-8-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
2016-12-19 20:50:43,575 [pool-8-thread-1] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
2016-12-19 20:50:43,598 [pool-8-thread-1] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 103 bytes
2016-12-19 20:50:43,614 [pool-8-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merged 1 segments, 110 bytes to disk to satisfy reduce memory limit
2016-12-19 20:50:43,615 [pool-8-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 1 files, 114 bytes from disk
2016-12-19 20:50:43,630 [pool-8-thread-1] INFO org.apache.hadoop.mapreduce.task.reduce.MergeManagerImpl - Merging 0 segments, 0 bytes from memory into reduce
2016-12-19 20:50:43,630 [pool-8-thread-1] INFO org.apache.hadoop.mapred.Merger - Merging 1 sorted segments
2016-12-19 20:50:43,631 [pool-8-thread-1] INFO org.apache.hadoop.mapred.Merger - Down to the last merge-pass, with 1 segments left of total size: 103 bytes
2016-12-19 20:50:43,635 [pool-8-thread-1] INFO org.apache.hadoop.mapred.LocalJobRunner - 1 / 1 copied.
2016-12-19 20:50:43,719 [Thread-50] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce task executor complete.
2016-12-19 20:50:43,759 [Thread-50] WARN org.apache.hadoop.mapred.LocalJobRunner - job_local1354877940_0003
java.lang.Exception: java.io.IOException: Mkdirs failed to create file:/home/soukaina/Bureau/results/_temporary/0/_temporary/attempt_local1354877940_0003_r_000000_0 (exists=false, cwd=file:/home/soukaina/Bureau)
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529)
Caused by: java.io.IOException: Mkdirs failed to create file:/home/soukaina/Bureau/results/_temporary/0/_temporary/attempt_local1354877940_0003_r_000000_0 (exists=false, cwd=file:/home/soukaina/Bureau)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:442)
at org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:428)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigTextOutputFormat.getRecordWriter(PigTextOutputFormat.java:98)
at org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:81)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:540)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:614)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:389)
at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:319)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2016-12-19 20:50:43,922 [main] WARN org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Ooops! Some job has failed! Specify -stop_on_failure if you want Pig to stop immediately on failure.
2016-12-19 20:50:43,986 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - job job_local1354877940_0003 has failed! Stop running all dependent jobs
2016-12-19 20:50:43,988 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - 100% complete
2016-12-19 20:50:44,081 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-19 20:50:44,096 [main] INFO org.apache.hadoop.metrics.jvm.JvmMetrics - Cannot initialize JVM Metrics with processName=JobTracker, sessionId= - already initialized
2016-12-19 20:50:44,102 [main] ERROR org.apache.pig.tools.pigstats.mapreduce.MRPigStatsUtil - 1 map reduce job(s) failed!
2016-12-19 20:50:44,104 [main] INFO org.apache.pig.tools.pigstats.mapreduce.SimplePigStats - Script Statistics:

HadoopVersion PigVersion UserId StartedAt FinishedAt Features
2.6.0 0.15.0 hduser 2016-12-19 20:50:39 2016-12-19 20:50:44 GROUP_BY

Failed!

Failed Jobs:
JobId Alias Feature Message Outputs
job_local1354877940_0003 A,B,C,D,E GROUP_BY,COMBINER Message: Job failed! file:///home/soukaina/Bureau/results,

Input(s):
Failed to read data from "file:///home/soukaina/Bureau/hdfile"

Output(s):
Failed to produce result in "file:///home/soukaina/Bureau/results"

Counters:
Total records written : 0
Total bytes written : 0
Spillable Memory Manager spill count : 0
Total bags proactively spilled: 0
Total records proactively spilled: 0

Job DAG:
job_local1354877940_0003


2016-12-19 20:50:44,132 [main] INFO org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher - Failed!
2016-12-19 20:50:44,252 [main] ERROR org.apache.pig.tools.grunt.GruntParser - ERROR 2244: Job job_local1354877940_0003 failed, hadoop does not return any error message
2016-12-19 20:50:44,269 [main] WARN org.apache.pig.tools.grunt.GruntParser - There is no log file to write to.
2016-12-19 20:50:44,270 [main] ERROR org.apache.pig.tools.grunt.GruntParser - org.apache.pig.backend.executionengine.ExecException: ERROR 2244: Job job_local1354877940_0003 failed, hadoop does not return any error message
at org.apache.pig.tools.grunt.GruntParser.executeBatch(GruntParser.java:179)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:234)
at org.apache.pig.tools.grunt.GruntParser.parseStopOnError(GruntParser.java:205)
at org.apache.pig.tools.grunt.Grunt.exec(Grunt.java:81)
at org.apache.pig.Main.run(Main.java:631)
at org.apache.pig.Main.main(Main.java:177)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
at org.apache.hadoop.util.RunJar.main(RunJar.java:136)

2016-12-19 20:50:44,604 [main] INFO org.apache.pig.Main - Pig script completed in 1 minute, 1 second and 532 milliseconds (61532 ms)
2016-12-19 20:50:49,372 [communication thread] INFO org.apache.hadoop.mapred.LocalJobRunner - reduce > reduce
0
Chercheur000 Messages postés 5 Date d'inscription dimanche 18 décembre 2016 Statut Membre Dernière intervention 20 décembre 2016
19 déc. 2016 à 21:58
si vous remarquez ça donne success pour les premières commandes
0
Réponse 2 / 2
mamiemando Messages postés 33108 Date d'inscription jeudi 12 mai 2005 Statut Modérateur Dernière intervention 14 mai 2024 7 753
20 déc. 2016 à 00:05
Ce n'est pas vraiment mon domaine, mais l'erreur est : 

Failed Jobs: 
JobId Alias Feature Message Outputs 
job_local1472906397_0003 A,B,C,D,E GROUP_BY,COMBINER Message: Job failed! file:///home/soukaina/Bureau/results,


Du coup peut être que parmi A,B,C,D,E l'un des jobs a planté, ou en peut être que le résultats ne sont pas homogène et ne permettent pas de faire le "GROUP BY". As-tu regardé si ces 5 variables contenaient le résultat attendu ?

Bonne chance
0
Chercheur000 Messages postés 5 Date d'inscription dimanche 18 décembre 2016 Statut Membre Dernière intervention 20 décembre 2016
Modifié par Chercheur000 le 20/12/2016 à 12:33
Oui ces 5 variables contiennent le résultat attendu.
en fait j'ai exécuté un programme pareil en mode mapreduce et tout a bien marché même la commande STORE
le problème reste au niveau de la commande STORE en mode local.

merci.
0
mamiemando Messages postés 33108 Date d'inscription jeudi 12 mai 2005 Statut Modérateur Dernière intervention 14 mai 2024 7 753
21 déc. 2016 à 12:01
Alors, ce n'est pas du tout ma spécialité, mais je vois le mot clé GROUP BY, COMBINER.

- Est-ce que cette primitive a du sens par rapport à chacuns des résultats observés dans A,B,C,D,E ?

- As-tu essayé de l'appliquer progressivement (à A, puis A,B, etc... pour localiser la cause du problème) ?
0

Discussions similaires

traduction francais - latin
michelium -
Latiniste -

6 réponses
Traducteur Latin -> Français
rvlf -
Loulou34300 -

12 réponses
traducteur français latin
roentgen -
Raymond PENTIER -

7 réponses
Traduction latin-francais
Mawa784 -
Mawa784 -

2 réponses
traduction latin francais
raroul -
raroul -

6 réponses