It can mess the filter if you use motor oil in place of transmission oil. This will hamper transmission. One should clean the pan and install a new filter. Remember to drain the motor oil first and flush it by adding the appropriate transmission oil. You may take the car to an instant lube for a check.
Yes, you will mess your transmission up, if you put motor oil in the spot for transmission oil. Transmission oil needs to be used always. Don't try and put in motor oil instead. If you accidentally do this, drain out the motor oil and put the transmission oil in.