java - Spring Batch partitioning a step -
i have multiple csv files read. want processing done 1 file @ time. rather reading records till reaches commit level.
i have put job uses partition on running job see there 2 entries every row. if job running twice.
<?xml version="1.0" encoding="utf-8"?> <beans xmlns="http://www.springframework.org/schema/beans" xmlns:context="http://www.springframework.org/schema/context" xmlns:p="http://www.springframework.org/schema/p" xmlns:batch="http://www.springframework.org/schema/batch" xmlns:mvc="http://www.springframework.org/schema/mvc" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance" xsi:schemalocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-4.0.xsd http://www.springframework.org/schema/context http://www.springframework.org/schema/context/spring-context-4.0.xsd http://www.springframework.org/schema/mvc http://www.springframework.org/schema/mvc/spring-mvc-4.0.xsd http://www.springframework.org/schema/batch http://www.springframework.org/schema/batch/spring-batch-2.2.xsd"> <import resource="classpath:/database.xml" /> <bean id="asynctaskexecutor" class="org.springframework.core.task.simpleasynctaskexecutor" > <property name="concurrencylimit" value="1"></property> </bean> <bean id="taskexecutor" class="org.springframework.scheduling.concurrent.threadpooltaskexecutor"> <property name="corepoolsize" value="5" /> </bean> <bean id="partitioner" class="org.springframework.batch.core.partition.support.multiresourcepartitioner" scope="step"> <property name="resources" value="file:#{jobparameters[filepath]}/*.dat" /> </bean> <bean id="multiresourcereader" class="org.springframework.batch.item.file.multiresourceitemreader" scope="step"> <property name="resources" value="file:#{jobparameters[filepath]}/*.dat"></property> <property name="delegate" ref="logitfilereader"></property> </bean> <batch:job id="remediationjob"> <batch:step id="partitionedstep" > <batch:partition step="readwritecontactspartitionedstep" partitioner="partitioner"> <batch:handler task-executor="asynctaskexecutor" /> </batch:partition> </batch:step> </batch:job> <batch:step id="readwritecontactspartitionedstep"> <batch:tasklet> <batch:transaction-attributes isolation="read_uncommitted"/> <batch:chunk reader="multiresourcereader" writer="rawitemdatabasewriter" commit-interval="10" skip-policy="pdwuploadskippolicy"/> <batch:listeners> <batch:listener ref="customitemreaderlistener"></batch:listener> <batch:listener ref="csvlineskiplistener"></batch:listener> <batch:listener ref="getcurrentresourcechunklistener"></batch:listener> </batch:listeners> </batch:tasklet> </batch:step> <bean id="logitfilereader" class="org.springframework.batch.item.file.flatfileitemreader" scope="step"> <!-- read csv file --> <property name="strict" value="false"></property> <property name="linemapper"> <bean class="org.springframework.batch.item.file.mapping.defaultlinemapper"> <!-- split --> <property name="linetokenizer"> <bean class="org.springframework.batch.item.file.transform.delimitedlinetokenizer"> <property name="delimiter" value="@##@" /> <property name="strict" value="true" /> </bean> </property> <property name="fieldsetmapper"> <!-- map object --> <bean class="org.kp.oppr.remediation.batch.vo.csvdatavofieldmapper"> </bean> </property> </bean> </property> </bean> <bean id="rawitemdatabasewriter" class="org.kp.oppr.remediation.batch.csv.rawitemdatabasewriter" scope="step"> </bean> <bean id="pdwuploadskippolicy" class="org.springframework.batch.core.step.skip.alwaysskipitemskippolicy" /> <bean id="csvdatavo" class="org.kp.oppr.remediation.batch.vo.csvdatavo" scope="prototype"></bean> <!-- batch listeners --> <bean id="pdwfilemoverlistener" class="org.kp.oppr.remediation.batch.listener.pdwfilemoverlistener" scope="step"> </bean> <bean id="csvlineskiplistener" class="org.kp.oppr.remediation.batch.listener.csvlineskiplistener" scope="step"> </bean> <bean id="customitemreaderlistener" class="org.kp.oppr.remediation.batch.listener.customitemreaderlistener"></bean> <bean id="getcurrentresourcechunklistener" class="org.kp.oppr.remediation.batch.listener.getcurrentresourcechunklistener"> <property name="proxy" ref ="multiresourcereader" /> </bean> <!-- <bean id="steplistener" class="org.kp.oppr.remediation.batch.listener.examplestepexecutionlistener"> <property name="resources" ref="multiresourcereader"/> </bean> --> <!-- skip policies --> </beans>
is there missing here ?
well have 2 questions there
1 - "i want processing done 1 file @ time. rather reading records till reaches commit level." set commit-interval 1 - read item, process , writer wait until has 1 item write.
2 - if job running twice.
looks run many times number of files have.
you should not use multiresourceitemreader step. partitioner splits resources multiple , create separate execution contexts. multiresourceitemreader again considering files because of resources property being set.
Comments
Post a Comment