IBM Mainframe Forum Index
 
Log In
 
IBM Mainframe Forum Index Mainframe: Search IBM Mainframe Forum: FAQ Register
 

What is maximum number records per a file is allowed in ftp


IBM Mainframe Forums -> All Other Mainframe Topics
Post new topic   Reply to topic
View previous topic :: View next topic  
Author Message
nareshdacha

New User


Joined: 12 Jan 2010
Posts: 66
Location: US

PostPosted: Fri May 27, 2011 4:22 am
Reply with quote

Hi,

Am trying to ftp around 51,000 records to mainframe host, here am getting the following error from the SYSLOG:

V370040 END OF VOLUME RECOVERY OPERATION STARTING FOR
V370040 FTPD2,STEP1,SYS00001,CSLT.NC.PDPC.EMAF.FILE
V370040 CURRENTLY ON EXTENT # 16 ON VOLUME TSS059
V370040 END OF VOLUME RECOVERY OPERATION SUCCESSFUL FOR
V370040 FTPD2,STEP1,SYS00001,CSLT.NC.PDPC.EMAF.FILE
V370040 NEW VOLUME TSS068 ADDED, NEW VOLUME COUNT 20

VAM-1606 DATASET NOW SPANS 20 VOLUMES
VAM-1606 POSSIBLE PROGRAM LOOP
VAM-1606 NO MORE VOLUMES WILL BE ADDED

What is maximum number records per a file can be ftp to mainframes ?
Please give your valuable inputs.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Fri May 27, 2011 5:08 am
Reply with quote

Quote:
What is maximum number records per a file can be ftp to mainframes ?
This question cannot be answered as stated. In order to know the maximum number of records, first you must know the RECFM, LRECL, and BLKSIZE. You did not provide us these values. You also have to indicate which release of z/OS you're running on and how big are the disk drives.

For an example, using 3390 mod 3 (3330 -- roughly -- usable cylinders each), fixed 80-byte records, half-track blocking:
Code:
Records per block                349
Records per track                698
Records per cylinder          10,470
Records per disk pack     34,865,100
Maximum records        2,057,040,900

(based on 59 disk volumes allowed per file)

The FTP messages are very clearly telling you that the file space allocation is extremely under-allocated for the number of records being placed in the file -- 20 volumes? However, as long as your site allows the network to do so, you could transfer a 2-billion record file. We routinely, every day, transfer 6 to 8 gigabyte files (millions of records) to and from the mainframe.

In other words, you are asking the wrong question. Rather than asking what the maximum limit is (51000 records is WAY under the limit), you should be asking why is your file transfer not working? How many bytes are being transferred before the problem occurs? How big are the records? What does the file look like when the FTP is done? There is much you should be doing to investigate this problem, and none of it we can do since we don't work at your site and don't have access to the data.
Back to top
View user's profile Send private message
enrico-sorichetti

Superior Member


Joined: 14 Mar 2007
Posts: 10872
Location: italy

PostPosted: Fri May 27, 2011 5:09 am
Reply with quote

the messages You received are not IBM messages

IIRC there is no theoretical limit in se, only a dasd space limit
but a limit on maximum amount of data that can be transferred can be set as a customization parameter

the best place to ask would be Your support!
both for the error You are receiving ( the VAM messages ) an the limit for data transfer
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Fri May 27, 2011 6:22 am
Reply with quote

Hello,

Which ftp product is being used?

Post the jcl and control statements used.

Code:
VAM-1606 DATASET NOW SPANS 20 VOLUMES
VAM-1606 POSSIBLE PROGRAM LOOP
VAM-1606 NO MORE VOLUMES WILL BE ADDED
Seems rather clear. . .
Back to top
View user's profile Send private message
nareshdacha

New User


Joined: 12 Jan 2010
Posts: 66
Location: US

PostPosted: Fri May 27, 2011 10:23 pm
Reply with quote

Thanks for response.
File Details:

Code:


  Volume serial . . . : TSS009
  Device type . . . . : 3390
 Data class . . . . . : DCBASE
  Organization  . . . : PS       
  Record format . . . : FB       
  Record length . . . : 121
  Block size  . . . . : 27951
  1st extent cylinders: 1
  Secondary cylinders : 100     
  Data set name type  :         
  SMS Compressible. . : NO       

Current Allocation
 Allocated cylinders : 1
 Allocated extents . : 1



FTP details:
Code:


FTP_TARGET_ADDR=XXXX
FTP_TARGET_USER=YYYY
FTP_INT_ADDR=ZZZZZ
FTP_INT_USER=YYYY
*
MB_IP_ADDRESS=XXX.com
MB_SERVICE_ID=YYYY
MB_FILE_PATH=/ABC/NEW PATH/SSSS/
SFTPUSER=YYYY
*


User name and password details are correct and it is able to login to the server but we got the below error messages in that step:
Code:

EZA1736I DIR 'dataset name'                 
EZA1701I >>> PORT <port no>
200 Port request OK.
EZA1701I >>> LIST 'dataset name'
550 No data sets found.
EZA1735I Std Return Code = 14550, Error Code = 00002
EZA1701I >>> QUIT
221 Quit command received. Goodbye.
EZA1736I EZAFTPLC -e (TIMEOUT 600 EXIT=12 TCP IBMTCPIP

Is there any way to know record count allowed for FTP based on the record length,format and space allocated for that dataset?
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Fri May 27, 2011 10:40 pm
Reply with quote

BLKSIZE 27951 divided by LRECL 121 gives 231 records per block. This is 462 records per track or 6,930 records per cylinder. Past that, it depends upon the space available on the volume. If the volume is nearly empty, you could get 15 secondary extents (100 cylinders each) plus the 1 cylinder primary, or 1501 cylinders total on a pack. This would allow for 10,401,930 records. If the volume does not have that much free space, however, the count of allowed records could be as low as 6930. 20 volumes means a minimum of 138,600 records since each volume will allocate the primary space of 1 cylinder so each volume.

Obviously, either the file has more than 51000 records in it, or the FTP process is looping for some reason, or the network is causing a glitch in the FTP, or your space allocation rules have a flaw, or secondary allocation is having a problem, or ....

If you truly think you have 51000 records, delete and redefine the file to have 8 cylinders primary space. This would allow the file to contain the entire transfer without going into secondary allocation.
Back to top
View user's profile Send private message
nareshdacha

New User


Joined: 12 Jan 2010
Posts: 66
Location: US

PostPosted: Wed Jun 01, 2011 1:58 am
Reply with quote

Thanks Robert & All.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Wed Jun 01, 2011 2:21 am
Reply with quote

Hello,

Is this now working?

Posting the resolution may help someone else with a similar situation one day. . .

d
Back to top
View user's profile Send private message
nareshdacha

New User


Joined: 12 Jan 2010
Posts: 66
Location: US

PostPosted: Wed Jun 01, 2011 3:09 am
Reply with quote

No Dick, Its not working. Still showing the same reason..
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Wed Jun 01, 2011 3:50 am
Reply with quote

Naresh, did you find where the messages come from? It seems like some sort of non-standard product. Do you have another ftp tool to try with?
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Wed Jun 01, 2011 7:03 am
Reply with quote

Hello,

Where is this ftp process running? Which ftp software is being used?

Has this ftp software ever been used successfully to transfer data to the mainframe?

Suggest you create a file of exactly 1 record and transmit this. Post all of the informatonal messages generated when this is run.
Back to top
View user's profile Send private message
nareshdacha

New User


Joined: 12 Jan 2010
Posts: 66
Location: US

PostPosted: Mon Jun 06, 2011 9:00 pm
Reply with quote

We are able to transmit the same kind of file on daily runs but on that day it got failed.
Based on the record length and blocksize, is there any way to calculate
how many records we can ftp ?

I did't understand how Robert came to conclusion about the below statements. Can anyone please clarify the below -
- This is 462 records per track or 6,930 records per cylinder
- This would allow for 10,401,930 records.
Back to top
View user's profile Send private message
Robert Sample

Global Moderator


Joined: 06 Jun 2008
Posts: 8696
Location: Dubuque, Iowa, USA

PostPosted: Mon Jun 06, 2011 9:17 pm
Reply with quote

Quote:
Based on the record length and blocksize, is there any way to calculate
how many records we can ftp ?
As stated before, this is not the right question to ask. The number of records you CAN transfer via FTP depends upon network parameters, site TCP/IP settings, record length, block size, space available on disk, number of volumes allocated to the data set, and so forth -- even settings on the other server may affect the number of records that you CAN transfer. FTP problems are almost NEVER due to hitting limits on the records to transfer but rather hitting another limit.

Quote:
Can anyone please clarify the below -
- This is 462 records per track or 6,930 records per cylinder
- This would allow for 10,401,930 records.
A 3390 has 15 tracks per cylinder. Not knowing a basic fact like this indicates you would be far, far better off on Beginner's and Student's Forum

Defining a file as SPACE=(CYL,(1,100)) would allow 1 primary and 15 secondary extents on a volume, or 1,501 cylinders. 1,501 cylinders times 6930 records yields 10,401,930 records. This is, again, basic data that you should know already.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Mon Jun 06, 2011 9:32 pm
Reply with quote

nareshdacha wrote:
We are able to transmit the same kind of file on daily runs but on that day it got failed.
[...]


Something in the data that day caused your problem, not anything to do with the number of records. The problem it caused was to send your FTP program into a loop, writing endlessly until it ran out of space.

You have to find the data that caused the problem, if you can. Once identified, you can avoid in future and raise a fault with the software supplier.

Why didn't you say the above originally?

On the source system, split the file into two equal parts. Send first, send second. If both are successful, join them again from the split and try to send the whole file (it is possible that the method you choose to do the split will "remove" the fault if you are unlucky).

If one fails, that is good (run it with 8 cyls of space and no secondary, so that it blows up quickly).

Same process on the broken segment. Continue until you have a manageable number of records to view yourself. Inspect file, identify error.

It won't take as long as it sounds, because of powers-of-two. Calculate. If you can eyeball 200 records, 400, 800, 1600, 3200, 6400, 12800, 25600, >5100. Nine splits or so.
Back to top
View user's profile Send private message
Bill Woodger

Moderator Emeritus


Joined: 09 Mar 2011
Posts: 7309
Location: Inside the Matrix

PostPosted: Mon Jun 06, 2011 9:45 pm
Reply with quote

Maybe even quicker. Run the transfer with data unchanged, space 8 cyls primary (thanks to Robert's calculation) and no secondary. When it blows, browse the file.

You will see either a repeated record, which will be the problem, or the one before, or the one after, or you will see data that looks OK. If it looks OK, then it is repeating a group of records, possibly the entire file. You would have to find where the repetition starts. Then find that on the input file from the source system. Presto.
Back to top
View user's profile Send private message
nareshdacha

New User


Joined: 12 Jan 2010
Posts: 66
Location: US

PostPosted: Mon Jun 06, 2011 10:26 pm
Reply with quote

Thanks for the clarification.
Back to top
View user's profile Send private message
madmartinsonxx

New User


Joined: 10 Dec 2010
Posts: 96
Location: Massachusetts

PostPosted: Tue Jun 07, 2011 2:23 am
Reply with quote

btw. is there an ftp manual for z/OS somewhere close by here please?
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Tue Jun 07, 2011 2:35 am
Reply with quote

Hello,

Start here:
publibz.boulder.ibm.com/cgi-bin/bookmgr_OS390/BOOKS/f1a1b950/CONTENTS?
Back to top
View user's profile Send private message
madmartinsonxx

New User


Joined: 10 Dec 2010
Posts: 96
Location: Massachusetts

PostPosted: Tue Jun 07, 2011 6:17 am
Reply with quote

thx, it does indeed help Dick.
Back to top
View user's profile Send private message
dick scherrer

Moderator Emeritus


Joined: 23 Nov 2006
Posts: 19244
Location: Inside the Matrix

PostPosted: Tue Jun 07, 2011 6:33 am
Reply with quote

You're welcome icon_smile.gif

d
Back to top
View user's profile Send private message
View previous topic :: :: View next topic  
Post new topic   Reply to topic View Bookmarks
All times are GMT + 6 Hours
Forum Index -> All Other Mainframe Topics

 


Similar Topics
Topic Forum Replies
No new posts How to split large record length file... DFSORT/ICETOOL 10
No new posts Extracting Variable decimal numbers f... DFSORT/ICETOOL 17
No new posts SFTP Issue - destination file record ... All Other Mainframe Topics 2
No new posts Access to non cataloged VSAM file JCL & VSAM 18
No new posts Compare only first records of the fil... SYNCSORT 7
Search our Forums:

Back to Top