CODEWITHSUNDEEP
gomapreduce
Aj002Th
Aj002Th

Reputation: 9

job count test in mit6.824 lab1-mapreduce, ioutil.ReadDir function get duplicate files

my code can pass all the test but not include the job count test in mit6.824 lab1-mapreduce.
output is as follow:

*** Starting job count test.
--- map jobs ran incorrect number of times (10 != 8)
--- job count test: FAIL

I run the job count section of test-mr.sh separately by commenting the rest of the file but found that I actually have 8 files not 10.
output file image

The map and reduce function of job count plugin in jobcount.go is as follow:

func Map(filename string, contents string) []mr.KeyValue {
    me := os.Getpid()
    f := fmt.Sprintf("mr-worker-jobcount-%d-%d", me, count)
    count++
    err := ioutil.WriteFile(f, []byte("x"), 0666)
    if err != nil {
        panic(err)
    }
    time.Sleep(time.Duration(2000+rand.Intn(3000)) * time.Millisecond)
    return []mr.KeyValue{mr.KeyValue{"a", "x"}}
}

func Reduce(key string, values []string) string {
    files, err := ioutil.ReadDir(".")
    if err != nil {
        panic(err)
    }
    invocations := 0
    for _, f := range files {
            invocations++
    }
    return strconv.Itoa(invocations)
}

test program for job count is as follow:

echo '***' Starting job count test.

rm -f mr-*

$TIMEOUT ../mrcoordinator ../pg*txt &
sleep 1

$TIMEOUT ../mrworker ../../mrapps/jobcount.so &
$TIMEOUT ../mrworker ../../mrapps/jobcount.so
$TIMEOUT ../mrworker ../../mrapps/jobcount.so &
$TIMEOUT ../mrworker ../../mrapps/jobcount.so

NT=`cat mr-out* | awk '{print $2}'`
if [ "$NT" -eq "8" ]
then
  echo '---' job count test: PASS
else
  echo '---' map jobs ran incorrect number of times "($NT != 8)"
  echo '---' job count test: FAIL
  failed_any=1
fi

wait

My environment is ubantu in wsl2.

i add some code in jobcount.go

for _, f := range files {
        if strings.HasPrefix(f.Name(), "mr-worker-jobcount") {
            log.Printf("f: %v\n", f)
            invocations++
        }
    }

and i find that i handle same file many times.
log file image

so i change the reduce function in jobcount.go to remove duplicate element.

func Reduce(key string, values []string) string {
    files, err := ioutil.ReadDir(".")
    if err != nil {
        panic(err)
    }
    invocations := 0

    mp := map[string]struct{}{}

    for _, f := range files {
        if _, ok := mp[f.Name()]; strings.HasPrefix(f.Name(), "mr-worker-jobcount") && !ok {
            invocations++
            mp[f.Name()] = struct{}{}
        }
    }
    return strconv.Itoa(invocations)
}

then i can pass all the test

i test and debug for many time but still have no idea about why it happen.

i want to know why ioutil.ReadDir function's output is wrong and how some poooible way i can pass the test without change the jobcount.go .

Upvotes: 0

Views: 129

Answers (1)

fcjcbk
fcjcbk

Reputation: 1

I have the same problem in wsl2, the reason may be you run the test in windows file system, I copy the test in wsl2's file system(eg. /home/user/), and all is fine.

Upvotes: -1

Related Questions